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I.  INTRODUCTION 


The  United  States  Marine  Corps  has  been  making  an  effort  to  improve  training 
with  the  use  of  simulation  and  virtual  environments.  The  intent  of  this  effort  is  to 
decrease  the  time,  personnel  and  materiel  resources  required  to  conduct  training;  improve 
accessibility,  and  reduce  complexity  in  order  to  increase  training  repetition  and  extend 
the  types  and  complexities  of  scenarios  that  can  be  trained.  In  the  January  2016,  in  the 
fragmentary  order  titled  Advance  to  Contact,  General  Robert  B.  Neller,  the  37th 
Commandant  of  the  Marine  Corps,  outlined  the  need  to  “capitalize  on  emerging 
technologies  and  future  opportunities  ...  to  leverage  a  virtual  and  constructive  training 
environment  and  tools”  (Department  of  the  Navy  United  States  Marine  Corps  [DoN 
USMC],  2016a,  p.  8)  and  “increase  the  amount  of  training  each  unit  can  accomplish  to 
‘increase  the  reps’  in  mentally  and  physically  stressing  environment  for  all  elements  of 
the  [Marine  Air  Ground  Task  Force  (MAGTF)]  before  they  do  so  on  the 
battlefield. ”(DoN  USMC,  2016,  p.  8).  The  September  2016  Marine  Corps  Operating 
Concept:  How  an  Expeditionary  Force  Operates  in  the  21st  Century  reiterates  this 
point  by  listing  the  need  to  “leverage  increased  computing  power  and  more  powerful 
simulation  tools  to  increase  the  number  of  turns,  repetitions,  and  scenarios  in  training” 
(DoN  USMC,  2016b,  Section  6.5.2). 

One  community  that  could  benefit  from  the  use  of  simulations  for  training  is  the 
Direct  Air  Support  Center  (DASC).  The  DASC  is  one  of  three  primary  air  control 
agencies  in  the  Marine  Corp’s  Marine  Air  Command  and  Control  System  (MACCS). 
Within  the  DASC,  air  support  control  officers  (ASCOs)  are  the  crew  members 
responsible  for  devising  safe  routing,  communing  routes,  air  hazards,  mission 
information,  and  friendly  and  enemy  disposition  to  pilots;  logging  routing  and  relevant 
mission  data  for  each  supported  mission,  and  reporting  requisite  information  to 
designated  MACCS  agencies.  ASCOs  play  an  important  part  in  facilitating  the  safe  and 
successful  conduct  of  air  operations  in  DASC-controlled  airspace.  They  are  also  critical 
in  integrating  air  and  ground  missions  in  support  of  MAGTF  operations.  The  focus  of  this 
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thesis  is  the  analysis  of  the  ASCO  task,  with  the  goal  of  developing  an  affordable 
prototype  simulation  capable  of  demonstrating  the  potential  to  be  used  as  an  ASCO 
training  system. 

A.  PROBLEM  STATEMENT 

Current  training  for  ASCO  is  inefficient,  inadequate  and  costly.  It  does  not  take 
advantage  of  increased  computing  power  and  simulation  tools.  ASCO  training  events 
currently  do  not  provide  trainees  with  the  opportunity  to  participate  in  multiple  trials. 
Furthermore,  in  its  current  form,  ASCO  training  occurs  infrequently,  resulting  in  the 
inability  to  maintain  skills,  increase  automaticity,  or  develop  expertise. 

B.  RESEARCH  QUESTIONS 

1.  What  is  the  feasibility  of  developing  an  automated  training  system  to 
support  individual  position  training  of  ASCO? 

2.  How  would  subject  matter  experts  consider  a  training  system  to  support 
individual  position  training  of  ASCOs  in  context  of  current  training 
procedures? 

C.  SCOPE  AND  LIMITATIONS 

This  thesis  focused  on  the  development  of  a  prototype  training  simulation  for 
ASCO  individual  position  training.  This  prototype  was  built  using  the  principles  and  best 
practices  from  training  simulations,  games  for  training,  the  science  of  human  learning, 
and  cognitive  load  theory.  It  also  investigated  the  utility  of  such  a  prototype  simulation 
for  ASCO  training. 

The  first  step  was  an  investigation  of  extant  literature  and  technologies.  The 
literature  review  focused  on  the  aforementioned  areas,  as  well  as  current  simulation 
platforms  associated  with  related  fields,  current  speech-to-text  and  text-to- speech 
technologies,  and  development  environments. 

The  second  was  developing  a  prototype  simulation  to  support  ASCO  individual 
position  training.  This  step  included  a  task  analysis  of  ASCO  positions  to  scope  the 
prototype  and  guide  development  process. 
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The  third  step  was  conducting  a  subject  matter  expert  (SME)  review  in  order  to 
determine  if  the  incorporated  technologies  are  effective  and  if  the  DASC  community  sees 
such  a  system  as  a  potential  solution  to  training  gaps.  The  information  gathered  was  also 
used  to  form  suggestions  for  future  systems  developments. 

D.  BENEFITS  OF  THIS  STUDY 

It  is  proposed  here  that  the  addition  of  an  ASCO  individual  position  trainer  would 

•  Increase  the  frequency  of  training  for  ASCOs. 

•  Decrease  the  time  it  takes  for  ASCOs  to  reach  qualification. 

•  Increase  the  readiness  of  Marine  Air  Support  Squadrons  (MASS). 

•  Improve  the  quality  of  overall  DASC  operations. 

E.  THESIS  ORGANIZATION 

The  next  section  of  this  thesis  is  the  background  section.  The  background  section 
introduces  DASC  operations,  qualification  requirements,  training,  and  training  shortfalls. 
The  literature  review  follows  the  background  section  and  examines  training  transfer  of 
simulations  for  training,  the  process  of  human  skill  acquisition  and  how  it  relates  to 
ASCO  training,  instructional  system  design,  synthetic  voice  production,  and  voice 
recognition.  After  the  literature  review  is  the  methods  section  in  which  the  methods  used 
to  complete  this  work  are  described.  The  final  three  sections  are  the  results  section,  the 
conclusion,  and  future  work. 
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II.  BACKGROUND 


In  order  to  understand  the  potential  benefits  and  requirements  of  an  ASCO 
individual  position  training  simulation,  one  must  first  understand  the  mission  of  the 
DASC  and  how  it  fits  in  to  the  MAGTF,  the  unique  way  the  DASC  conducts  operations, 
and  the  role  the  ASCO  plays  within  the  DASC  crew. 

The  DASC’s  mission  consists  of  four  parts  that  provide  the  flexible  air  control 
and  support  necessary  to  offset  the  inherent  complexity  of  war.  The  first  part  of  the 
mission  is  to  procedurally  control  all  aircraft  in  DASC  controlled  airspace  (Department  of 
the  Navy  United  States  Marine  Corps  [DoN  USMC],  2010).  “Procedural  control  is  a 
method  of  airspace  control  that  relies  on  a  combination  of  previously  agreed  upon  and 
promulgated  orders  and  procedures”  (DoN  USMC,  2010,  p.  137).  The  DASC  uses 
procedural  control  exclusively  and  has  no  organic  electronic  means  of  tracking  aircraft. 
In  turn,  the  DASC  can  deploy  rapidly  in  austere  environments  with  a  relatively  small 
footprint. 

The  second  part  of  the  mission  is  to  process  immediate  air  support  requests  (DoN 
USMC,  2010).  This  means  that  the  DASC  maintains  communications  with  ground  units 
throughout  the  battle  space.  If  a  ground  unit  requires  unplanned  air  support,  such  as  a 
casualty  evacuation,  it  contacts  the  DASC  to  request  immediate  air  support.  The  DASC 
then  makes  a  decision  or  recommendation,  depending  on  their  authority  during  the 
operation,  on  how  and  if  the  request  is  supported  by  air.  If  the  request  is  supported,  the 
DASC  tasks  an  aircraft  with  supporting  the  request  and  passes  all  relevant  mission  data  to 
both  the  supporting  aircraft  and  the  supported  ground  unit. 

The  third  part  of  the  DASC  mission  is  to  integrate  aircraft  employment  with  other 
supporting  arms  (  DoN  USMC,  2010).  To  this  end,  the  DASC  either  is  collocated  with 
the  senior  fire  support  coordination  center  (FSCC)  or  provides  the  FSCC  with  an  air 
support  liaison  team  (ASLT).  This  provides  the  DASC  with  nearly  immediate  knowledge 
of  fire  missions  and  helps  them  to  facilitate  timely  fires  integration  while  maintaining 
safety  of  flight  for  all  aircraft.  For  instance,  if  a  ground  unit  were  in  contact  with  enemy 
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troops  and  requested  artillery  fire  in  support  of  a  maneuver,  the  DASC  would  clear  the 
airspace  along  the  line  of  fire  between  the  maximum  and  minimum  ordinate  of  the  fire 
mission  and  then  report  that  the  airspace  was  clear  to  the  FSCC  so  that  firing  could 
commence. 

The  fourth  part  of  the  DASC  mission  is  to  manage  terminal  control  assets  (DoN 
USMC,  2010).  When  aircraft  are  supporting  ground  operations,  often  a  terminal 
controller  is  responsible  for  the  direction  of  aircraft  to  deliver  ordnance,  passengers  or 
cargo  to  a  specific  location.  Terminal  control  often  requires  visual  tracking  of  the  aircraft 
and  identification  of  targets  or  landing  zones  (DoN  USMC,  2010).  These  operations  take 
place  in  DASC  controlled  airspace.  The  DASC  allocates  the  terminal  controller  a  block 
of  airspace  and  follows  hand-over  procedures  to  pass  aircraft  control  to  the  terminal 
control.  Depending  on  the  type  of  terminal  control,  the  DASC  may  allocate  radio 
frequencies  for  the  terminal  controller  to  communicate  with  the  aircraft.  The  DASC 
maintains  communications  with  terminal  controllers.  This  link  improves  coordination  and 
provides  the  terminal  controller,  and  thereby  the  supported  ground  unit,  a  means  to 
communicate  with  the  MACCS. 

In  order  to  accomplish  its  mission,  several  continuous  and  concurrent  tasks  must 
take  place.  These  tasks  are  carried  out  and  managed  by  a  crew  composed  of  both 
commissioned  and  non-commissioned  Marines.  The  crew  positions  include  (numbers  in 
parenthesis  indicate  a  DASC’s  normal  complement): 

1.  Senior  air  director  (SAD)  (1); 

2.  Crew  chief  (CC)  (1); 

3.  Air  support  control  officers  (ASCO)  (2): 

a.  Tactical  air  director  (TAD) 

b.  Helicopter  director  (HD) 

4.  Air  support  net  operators  (ASNO)  (multiple); 

5.  Plotters  (multiple)  (Department  of  the  Navy  United  States  Marine  Corps 

[DoN  USMC],  2015). 
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The  SAD  and  CC  are  the  most  qualified  commissioned  and  non-commissioned 
watch-standers,  respectively,  and  are  responsible  for  overall  operations  as  well  as  inter- 
and  intra-agency  coordination.  Several  ASNOs  are  responsible  for  communications  with 
other  command  and  control  agencies,  ground  units,  terminal  controllers,  and  DASC 
extensions  and  liaisons.  The  plotters  are  responsible  for  maintaining  DASC  displays  such 
as  the  map  and  the  radio  in  and  out  (RIO)  board — a  table  displaying  the  mission  data  of 
all  of  the  aircraft  controlled  by  the  DASC.  The  ASCOs  are  commissioned  officers  whose 
responsibilities  include 

1.  Providing  safe  routing,  mission  updates  and  threat  briefs  to  fixed  wing  and 
rotary  wing  aircraft; 

2.  Maintaining  status  and  mission  information  on  aircraft  under  their  control; 

3.  Coordinating  with  the  other  ASCO  to  ensure  there  is  no  routing  conflicts; 

4.  Reviewing  immediate  requests  and  providing  support  recommendations  to 
the  SAD." 

DASC  operations  are  fast-paced  and  complex.  ASCOs  require  a  broad  range  of 
knowledge  and  skills,  most  of  which  are  highly  perishable  such  as 

1.  Familiarity  with  both  ground  and  aviation  operations  and  terminology; 

2.  Knowledge  of  the  capabilities  and  limitation  of  organic,  joint,  and 
multinational  aircraft  and  aircraft  ordinance; 

3.  The  ability  to  communicate  quickly  and  effectively  via  radio; 

4.  The  ability  to  maintain  a  three  dimensional  mental  picture  of  the  airspace 
including  aircraft  locations; 

5.  The  ability  to  maintain  situational  awareness  of  the  rest  of  the  DASC 
crew’s  actions; 

6.  The  ability  to  multi  task  in  order  to  maintain  constant  communication  with 
aircraft,  update  logs,  and  report  required  information  to  the  rest  of  the 
command  and  control  system. 

ASCOs  are  required  to  be  qualified  in  two  positions — the  HD  and  TAD.  In  order 
to  earn  each  qualification,  ASCOs  must  meet  several  prerequisites.  First,  the  ASCO  must 
complete  air  support  control  officers  course  (ASCOC).  Next,  they  must  be  observed 
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conducting  the  duties  of  an  ASCO  in  a  DASC  at  operational  tempo  levels  one  through 
three  (Table  1).  Then,  they  must  be  deemed  proficient  by  a  senior  instructor  while 
operating  in  a  DASC  at  operational  tempo  levels  one  through  five.  Proficiency  is  defined 
as  correct,  efficient,  and  skillful  execution  of  tasks  without  hesitation,  requiring  minimal 
input  from  the  instructor  and  with  minor  errors  corrected  by  the  trainee.  Finally,  they 
must  be  deemed  proficient  by  a  weapons  and  tactic  instructor  (WTI)  at  operational  tempo 
levels  one  through  five  (DoN  USMC,  2013).  This  is  an  abbreviated  description  of  the 
qualification  requirements  focusing  on  practical  skills  and  does  not  include  academics, 
environmental  requirements  and  various  prerequisites.  A  full  description  of  requirements 
can  be  found  in  the  Direct  Air  Support  Center  Training  and  Readiness  Manual 
[NAVMC  3500.120]. 


Table  1.  Operational  Tempo  per  Hour  Requirements. 

Adapted  from  DoN  USMC  (2013). 


Level 

Fixed 

Wing 

Rotary 

Wing 

Immediate 

Requests 

Fire 

Missions 

1 

2 

2 

2 

1 

2 

4 

4 

3 

2 

3 

6 

6 

4 

3 

4 

8 

8 

5 

4 

5 

10 

10 

6 

5 

A.  CURRENT  TRAINING 

ASCOs  receive  initial  training  at  ASCOC  at  the  Marine  Corps  Communication- 
Electronics  School  (MCCES)  in  Twentynine  Palms,  California.  The  instruction  at 
ASCOC  is  comprised  of  a  classroom-based  lecture  series  that  provides  the  foundational 
education  required  to  understand  the  mission,  tasks,  coordination  and  terminology 
intrinsic  to  the  DASC  as  well  as  practical  application  to  train  and  test  the  skill  required 
during  DASC  operations.  ASCOC  provides  practical  experience  by  conducting  live 
simulations.  A  common  format  for  live  DASC  simulations  is  to  deploy  and  operate  two 
DASCs.  One  DASC  is  simulation  control  and  the  personnel  staffing  it  act  as  the  pilots, 


8 


MACCS  agencies,  DASC  extension,  and  ground  unit  with  which  the  DASC  coordinates. 
The  second  DASC  is  manned  by  the  training  audience.  They  assume  the  crew  positions 
on  which  they  are  being  trained  or  evaluated.  At  ASCOC,  each  session  lasts  twenty 
minutes  and  can  only  accommodate  two  ASCOs,  one  in  the  position  of  HD  and  one  in  the 
position  of  the  TAD.  The  rest  of  the  students  conduct  self  and  partner  study  while  they 
wait  between  four  and  six  hours  for  their  turn.  Self-study  generally  involves  reviewing 
published  coordination  procedures,  maps,  airspace  control  measures  (ACM)  and  the  air 
tasking  order  (ATO).  During  partner  study,  two  trainees  take  turns  acting  as  a  pilot  and 
an  ASCO.  They  reference  a  map  depicting  the  airspace  and  practices  routing  the  aircraft 
with  the  use  of  proper  radio  in/out  (RIO)  procedures. 

Training  at  a  MASS  (a  fleet  DASC  unit),  is  conducted  in  the  same  basic  format  as 
at  ASCOC.  Classroom  instruction  and  self-study  are  the  primary  means  of  knowledge 
expansion  and  live  DASC  simulations  and  partner  study  are  used  to  train  skill.  Outside  of 
these  simulations,  there  is  little  formal  training,  meaning  that  on  the  job  training  during 
live  exercises  and  in  support  of  military  operations  is  a  common  occurrence. 

B.  TRAINING  SHORTFALLS 

There  are  several  problems  with  these  training  methods.  The  on-the-job  training 
provided  by  exercises  rarely  reaches  an  operational  tempo  that  can  count  towards 
qualifications.  Also,  during  exercises,  the  DASC  is  not  the  primary  training  audience,  and 
many  of  the  agencies  with  which  the  DASC  doctrinally  coordinates  do  not  participate  or 
are  not  focused  on  training  that  requires  coordination  with  the  DASC.  Furthermore,  the 
airspace  available  for  the  procedural  control  of  aircraft  is  extremely  limited  and  cannot 
support  the  volume  of  aircraft  or  complexity  of  airspace  required  to  train  an  ASCO. 

Live  simulations  are  time  and  resources  intensive.  Coordinating  activities 
associated  with  planning  and  scenario  building,  resources  acquisition,  and  personnel  and 
material  availability  combine  to  result  in  difficult  and  complex  preparations.  For  this 
reason,  training  drills  are  infrequent,  often  taking  place  only  once  or  twice  a  year,  and 
only  last  for  a  couple  of  days.  Their  rarity  and  limited  duration  make  it  unlikely  that  all 
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personnel  requiring  training  will  be  able  to  participate.  Furthermore,  due  to  the  limited 
operational  tempo  reached  during  exercises,  these  drills  are  often  used  for  qualification 
instead  of  training.  This  means  that  trainees  are  expected  to  be  ready  to  be  evaluated 
having  received  little  to  no  simulation  time  beforehand. 

Live  simulations  focus  on  not  only  individual  training  but  also  crew  training  since 
all  of  the  crew  positions  interact.  These  interactions,  paired  with  the  inherently  adaptable 
form  of  airspace  control  performed  by  the  DASC,  make  live  simulations  complex.  Live 
simulations  generally  start  out  with  a  script  that  the  actors  use  to  provide  inputs.  In  a  live 
DASC  simulation,  it  is  unlikely  that  the  trainee  will  make  decisions  that  align  with  the 
prepared  script,  which  causes  deviations  and  changes  the  scenario.  It  is  up  to  a  training 
confederate  to  track  these  deviations  and  respond  accordingly  for  the  duration  of  the 
simulation  to  maintain  the  realism  of  the  training.  If  the  confederates  forget  a  change 
made  by  the  ASCO  and  report  an  aircraft  in  the  wrong  position,  it  is  not  just  a  break  from 
reality,  but  it  corrupts  the  mental  model  of  the  ASCO. 

Other  problems  exist  upon  completion  of  a  training  scenario.  For  example,  after 
action  playbacks  are  not  available.  This  leaves  instructors  little  choice  but  to  conduct 
performance  reviews  and  assessments  that  are  largely  subjective.  There  is  no 
standardized  means  of  evaluating  performance.  Usually,  a  senior  Marine  will  observe  and 
provide  feedback,  but  most  of  the  senior  Marines  are  on  the  crew  acting  as  the  SAD  or 
providing  inputs  to  the  training  exercise.  This  limited  amount  of  training  and  feedback  is 
detrimental  to  the  maintenance  or  improvement  of  the  skills  required  of  an  ASCO.  In 
summation,  numerous  critical  training  gaps  exists  which  can  impact  the  performance  of 
ASCOs,  the  DASC,  and  ultimately,  the  combat  effectiveness  of  the  MAGTF. 
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III.  LITERATURE  REVIEW 


It  is  not  enough  to  say  the  military  leadership  has  decided  to  use  simulation  for 
training  so  everybody  start  using  simulation.  There  must  be  evidence  that  the  addition  of 
a  training  simulation  will  result  in  better  trained  personnel  and  a  rigorous  means  of 
developing  systems  to  meet  specific  training  goals.  This  section  reviews  relevant 
literature  associated  with  the  transfer  of  training  from  simulation  and  virtual  based 
training  systems,  skill  acquisition,  and  instructional  system  design. 

A.  TRAINING  TRANSFER 

The  main  goal  of  any  training  is  the  transfer  of  targeted  knowledge,  skills,  and 
attitudes  to  real  world  performance  (Department  of  Defense  [DOD],  2001).  There  is 
significant  evidence  that  suggests  using  simulation  and  virtual  environments  for  training 
can  result  in  improved  performance.  For  example,  simulation-aided  performance 
improvement  was  shown  during  the  Royal  Australian  Air  Forces  live  training  event,  Pitch 
Black  (Francis,  Best  &  Yildiz,  2016).  Prior  to  Pitch  Black,  a  group  of  trainees 
participated  in  a  simulation  supported  training  exercise  called  Black  Skies.  The  intent 
behind  Black  Skies  was  to  prepare  trainees  for  the  more  expensive  live  training  of  the 
Pitch  Black  exercise.  Results  demonstrated  a  twenty  percent  average  improvement  in 
performance  during  Pitch  Black  by  those  teams  that  participated  in  Black  Skies  compared 
to  those  that  did  not  (Francis  et  al.,  2016).  Another  study  evaluated  the  effectiveness  of  a 
virtual  training  system  called  the  flooding  control  trainer  (FCT).  Conducted  at  Naval 
Recruit  Training  Command,  the  FCT  experiment  demonstrated  a  50%  reduction  in 
decision-making  errors,  an  80%  reduction  in  communication  errors,  and  a  50% 
improvement  in  situational  awareness  and  navigation  skills  during  the  training  capstone 
evaluation  called  Battle  Station  21  (Hussain  et  al.,  2009).  The  proposed  training  system 
for  the  FCT  study  used  a  computer  game  design  to  keep  learners  engaged  and  drive  home 
the  consequence  of  their  actions.  In  another  study  of  medical  residents’  ability  to  perform 
a  colonoscopy,  researchers  found  that  those  who  received  training  on  a  virtual  reality 
simulator  prior  to  the  evaluation  performed  significantly  better  than  those  who  received 
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only  conventional  training  (Park  et  al.  2006).  While  just  a  sample,  these  studies  come 
from  different  professions,  incorporated  different  types  of  training  systems,  and  trained 
different  skills  to  different  populations.  However,  each  example,  and  numerous  others, 
demonstrates  that  training  with  the  aid  of  simulation  and  virtual  environments  can  result 
in  real  world  performance  improvements. 

B.  SKILL  ACQUISITION 

As  discussed  in  Chapter  I,  there  are  several  reasons  why  the  DOD  and  the  Marine 
Corps  are  increasing  investments  in  simulation  and  virtual  training.  These  approaches  to 
training  have  been  shown  to  be  less  expensive  than  live  training,  able  to  allow  more 
complex  and  dangerous  mission  training,  and  increase  the  opportunities  for  training.  As  a 
stated  goal  of  this  thesis,  increasing  the  number  of  training  events  an  individual  ASCO 
can  perform  is  needed  to  improve  proficiency  and  readiness.  The  following  section 
addresses  why  repetition  is  important  and  how  the  benefits  of  such  an  increase  in  training 
event  could  be  advantageous  to  the  Marine  Corps. 

The  answer  to  why  increased  practice  is  advantageous  is  founded  in  several 
cognitive  theories.  The  theory  of  human  information  processing,  introduced  by  Atkinson 
and  Shiffrin  (1968),  examines  how  humans  take  in  information  and  process  it  into 
memory.  Under  this  theory,  stimuli  activate  the  sensory  register,  and  if  attention  is  given 
to  the  stimulus,  the  information  is  transferred  to  a  short-term  store,  or  working  memory, 
and  the  long-term  store  is  scanned  for  existing  relevant  information.  Upon  rehearsal  and 
association  with  prior  existing  knowledge,  the  information  in  the  short-term  register  is 
transferred  to  the  long-term  register.  Atkinson  and  Shiffrin  (1968)  suggest  that  the 
information  held  in  the  short-term  store  begins  to  decay  after  15-30  seconds  if  not 
rehearsed  and  transferred  to  the  long-term  resister,  which  they  describe  as  relatively 
decay  free. 

Short-term  memory  is  not  only  limited  by  duration  but  also  by  capacity.  Miller 
(1956)  proposed  that  people  are  only  able  to  hold  seven  plus  or  minus  two  chunks  of 
information  in  short-term  memory.  Even  though  the  number  of  chunks  is  relatively  static, 
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the  size  of  the  chunks  is  not.  As  a  person  becomes  more  familiar  with  the  information  he 
or  she  is  handling,  the  person  is  able  to  group  several  chunks  together.  This  group  is  then 
a  single  chunk,  and  the  total  amount  of  information  that  can  be  held  in  short-term 
memory  is  increased. 

Clark,  Nguyen,  and  Sweller  (2006)  liken  these  larger  capacity  chunks  to  schemas. 
Schema  is  one  of  many  terms  for  knowledge  that  is  stored  in  a  relational  network 
(Harper-Sciarini,  2010).  According  to  Clark  et  al.  (2006),  expanding  one’s  schemas  is 
one  way  to  reduce  the  impact  extraneous  load.  Cognitive  load  theory  finds  its  basis  in  the 
theory  of  human  information  processing.  As  discussed  before,  sort-term  memory  is  of 
limited  capacity  and  duration.  If  you  try  to  hold  too  many  chunks  in  short-term  memory, 
the  process  becomes  overloaded  and  capacity  is  diminished.  Eventually,  continuous 
overload  can  result  in  a  complete  shutdown  (Clark,  Nguyen  &  Sweller,  2006). 

Cognitive  load  comes  in  three  different  forms:  intrinsic,  which  is  load  due  to  the 
nature  of  the  task;  germane,  which  is  load  due  to  the  innate  diversity  of  the  task;  and 
extraneous,  which  is  load  that  is  irrelevant  to  the  task  (Mayer,  2014).  Of  these, 
instructional  designers  want  to  increase  germane  load  because  “germane  cognitive  load 
that  contributes  to,  rather  than  interferes  with,  learning”  (Sweller,  Van  Merrienboer  & 
Pass,  1998,  p.  264). 

As  a  simple  example  of  these  loads,  consider  an  addition  problem.  The  intrinsic 
load  would  increase  if  adding  multiple  figures  (ex.  3+3+3)  was  required,  germane  load 
would  come  from  doing  multiple  examples  (ex.  1+2+3,  5+9+2),  and  extraneous  load 
would  come  from  the  fact  that  each  number  that  is  being  added  is  on  a  different  page  so 
that  the  tasks  requires  flipping  through  a  work  book  to  find  each  number.  Ultimately,  our 
goal  is  to  decrease  the  extraneous  load  so  that  limited  mental  resources  can  be  spent 
dealing  with  intrinsic  and  germane  load.  Developing  larger  and  more  complex  schemas 
through  experience  and  practice  (assuming  the  appropriate  items  were  being  rehearsed) 
will  provide  the  means  to  relate  more  information  in  to  groups  and  ultimately  increase  the 
size  of  chunks  and  held  in  short-term  memory  and  decrease  the  likelihood  of  cognitive 
overload. 
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Another  way  to  manage  extraneous  load  is  to  build  automaticity.  Clark  et  al. 
(2006)  described  automaticity  as  the  rehearsal  of  skill  so  often  they  become  automatic. 
This  means  that  mental  processes  are  no  longer  being  spent  thinking  thorough  how  to 
accomplish  a  task.  Instead,  the  task  is  accomplished  automatically  and  mental  processes 
are  freed  to  deal  emerging  information. 

The  job  of  an  ASCO  is  cognitively  tasking.  Not  only  do  they  continuously  relay 
information  back  and  forth  with  pilots,  they  communicate  that  information  with  the  rest 
of  the  MACCS,  maintain  RIO  logs,  supervise  the  RIO  board  plotter,  track  updates  to 
aircraft  mission  data,  and  maintain  situational  awareness  of  changes  to  the  airspace. 
Furthermore,  since  the  DASC  does  not  have  radar,  they  accomplish  these  tasks  without 
live  visual  cues  of  aircraft  flight  information.  This  means  ASCOs  must  create  a  three- 
dimensional  model  in  their  heads  that  incorporates  the  routing  they  assigned  to  each 
aircraft  in  the  airspace.  Increasing  the  availability  of  relevant  practice  could  result  in 
enhance  schemas  allowing  them  to  deal  with  larger  amounts  of  information  and  more 
complex  scenarios  and  develop  automaticity  of  certain  skills  thereby  freeing  more  space 
in  working  (short-term)  memory. 

C.  INSTRUCTIONAL  SYSTEM  DESIGN 

Because  the  proposed  prototype  for  this  thesis  is  intended  to  be  a  training 
simulation,  it  is  imperative  to  consider  instructional  system  design  (ISD).  The  approach 
discussed  below  offers  a  framework  in  which  ISD  can  be  accomplished. 

The  aim  of  ISD  is  to  use  the  principles  of  learning  to  develop  learning  plans, 
resource  material,  activities  and  evaluation  that  results  in  efficient  learning  and  measured 
outcomes  (Hussain  &  Coleman,  2014).  While  there  are  several  different  models  for  ISD, 
the  preferred  approach  for  the  DOD  is  the  five  phase  analysis,  design,  development, 
implementation,  and  evaluation  (ADDIE)  model  (DOD,  2001).  The  analysis  phase 
focuses  on  what  learners  need  to  learn  and  who  the  learners  are.  This  is  often  done  by 
conducting  a  task  analysis  and  identifying  what  tasks  are  not  being  trained  by  current 
methods.  During  the  design  phase,  the  outcomes  of  the  analysis  and  appropriate  learning 
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theories  are  used  to  create  a  plan  to  produce  instruction  that  will  result  in  successful 
learning.  This  can  include  decomposing  complex  tasks  in  to  separate  lessons,  including 
feedback  to  support  learning,  and  developing  pass  /  fail  criteria.  The  development  phase 
is  closely  linked  with  design  and  constitutes  the  actual  production  of  prototypes  and  the 
final  system  and  associated  materials  that  support  the  system  and  training. 
Implementation  is  the  actual  delivery  of  the  system  to  the  learners.  Evaluation  is,  in 
theory,  continuous,  and  should  exist  at  every  level  of  the  process.  This  thesis  focused  on 
the  analysis,  design,  development,  and  evaluation  stage  of  the  ADDIE  process. 

D.  SYNTHETIC  VOICE  PRODUCTION  AND  VOICE  RECOGNITION 

The  full  results  of  the  ADDIE  model  are  given  in  the  results  section,  but  one 
factor  that  emerged  from  it  was  the  requirement  for  users  to  interact  with  the  system  via 
voice.  In  creating  a  simulation  that  relies  on  verbal  communication,  it  is  important  to 
identify  key  areas  that  can  be  automated  in  order  to  reduce  the  need  for  instructors  and 
operators  to  perform  as  training  confederates  (providing  pilot  communication  for  the 
trainees)  instead  of  their  primary  responsibilities.  For  example,  the  procedural  controls 
practiced  in  the  DASC  requires  constant  communication  between  pilots  and  the  ASCO. 
Because  of  this  persistent  two-way  communication,  current  training  requires  a  role 
player,  usually  a  senior  ASCO,  to  play  the  part  of  pilots  in  the  DASCs  area  of 
responsibility.  Role  players  are  also  required  for  every  other  DASC  crew  position  when 
conducting  training.  A  minimum  of  six  role  players  is  needed  to  conduct  training  for  a 
single  DASC  crew.  In  order  to  simulate  these  exchanges,  some  method  of  vocal 
interaction  between  the  trainee  and  the  system  was  necessary.  For  the  verbal  output  from 
the  system,  there  were  two  options:  a  prerecorded,  scripted,  solution,  or  dynamic, 
synthetic  voice  generation.  For  interpreting  the  trainee’s  speech,  the  system  required 
some  form  of  speech  recognition. 

1.  Computer  Produced  Verbal  Output 

When  creating  a  simulation  that  provides  verbal  stimuli  to  a  trainee,  the  stimuli 
can  be  produced  in  two  ways:  it  can  be  either  prerecorded  audio  clips  or  synthetic  voice 
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production.  Prerecorded  audio  has  humans  read  words  and  phrases  designers  anticipate 
the  system  will  have  to  say  to  the  user.  Synthetic  voice  production  is  the  manufacturing 
of  human  speech  via  electronic  means. 

Even  though  prerecorded  inputs  offer  better  quality,  they  tend  to  be  more 
expensive  and  less  adaptable.  Training  simulators  that  use  prerecorded  voices  would  only 
have  access  to  those  discrete  inputs  and  could  not  verbalize  any  comment  the  designers 
had  not  anticipated.  If  there  was  a  need  to  train  for  a  scenario  or  in  an  environment  not 
accounted  for  by  the  recording,  new  recordings  would  have  to  be  commissioned  and  the 
system  would  have  to  be  adapted  to  handle  new  inputs.  This  type  of  static  solution  would 
not  support  one  of  the  key  characteristics  of  DASC  operations,  adaptability.  Since  war  is 
inherently  chaotic  and  unpredictable,  the  Marine  Corps  operates  under  the  philosophy  of 
centralized  command  and  decentralized  control  in  order  to  maintain  the  flexibility  to 
respond  to  dynamic  environment.  The  DASC  is  no  different.  The  ASCO  controls  the 
airspace  using  general  guidelines  but  can  ultimately  do  whatever  is  needed  to  ensure  the 
safe  transit  of  aircraft.  A  system  that  uses  prerecorded  audio  would  not  be  able  account 
for  every  eventuality  and  could  even  provide  negative  training  by  artificially  restricting 
ASCO  actions. 

There  are  two  main  approaches  to  speech  synthesis,  formant-based,  also  called 
text  to  speech  by  rule,  and  waveform,  also  called  concatenative  synthesis.  The  formant- 
based  approach  uses  an  electronic  model  to  synthesize  the  “voice.”  This  method  was  used 
often  for  early  text  to  speech  technologies  and  focused  on  individual  segments  and  their 
acoustic  quality  (Winters  &  Pisoni,  2003).  Waveform,  on  the  other  hand,  is  a  newer 
technology  and  uses  a  library  of  prerecorded  sound  units  taken  from  real  human  speech. 
These  sound  units  are  concatenated  together  to  create  the  generated  “voice.”  The  focus  on 
this  method  is  on  the  transition  between  segments  (Winters  &  Pisoni,  2003).  The  sound 
units  in  the  waveform  method  are  often  diaphones,  “the  section  of  speech  from  the 
middle  of  one  phone  to  the  middle  of  the  next  phone”  (Cunningham,  2011,  p.  3),  though 
any  size  of  unit  can  be  used. 
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Cunningham  (2011)  explains  that  early  synthetic  voice  production  focused  on  the 
intelligibility  of  individual  words  or  segments,  as  is  seen  with  formant -based  synthesis, 
and,  more  recently,  the  focus  has  shifted  to  making  synthetic  voice  sound  more  natural 
and  encompassing  the  prosodic  cues  found  in  real  human  speech,  such  as  tempo,  rhythm, 
stress,  and  intonation.  Waveform  seeks  to  achieve  this  by  providing  smoother  transitions. 
As  technology  has  progressed,  so  too  has  the  prosodic  ability  of  waveform-based 
systems.  Higher  quality  synthesizers  and  the  use  of  natural  language  processors  have 
improved  prosody  by  adjusting  pitch  based  on  punctuation  and  the  presence  of 
homophones.  Also  common  in  higher  quality  synthesizers  is  the  use  of  a  large  number  of 
prerecorded  sound  units,  including  recordings  with  different  pitch.  This  gives  speech 
algorithms  more  samples  to  choose  from,  and  ultimately,  the  ability  to  produce  smoother 
transitions  (Cunningham,  2011). 

Another  important  area  of  research  is  the  effect  of  noise  on  the  intelligibility  of 
synthetic  speech.  This  is  of  particular  interest  in  the  case  of  a  training  simulation  for 
ASCOs  because  the  DASC  is  a  noisy  environment.  Since  any  well-designed  training 
system  should  be  extensible  (Zyda,  2001),  allowing  for  the  integration  of  other  crew 
positions  would  result  in  the  need  to  include  additional  noise  and  pose  another  challenge 
for  a  representative  system.  Pisoni  and  Koen  (1982)  found  that  there  was  an  interaction 
between  the  use  of  synthetic  voice  and  the  addition  of  white  noise.  Additionally,  as  the 
sound-to-noise  ratio  increased,  the  intelligibility  of  synthetic  segments  degrades  at  a 
higher  rate  than  natural  voice  (Winters  &  Pisoni,  2003).  When  investigating  babble 
instead  of  white  noise,  Koul  and  Allen  (1993)  demonstrated  that  the  addition  of  babble 
did  not  produce  an  interaction  with  synthetic  voice,  but  instead  degraded  synthetic  and 
natural  voice  evenly.  They  postulated  that  the  difference,  in  addition  to  some  specific 
degradation  of  the  constant  “h,”  may  arise  from  the  tendency  of  babble  to  remain  at  a 
lower  frequency  while  white  noise  is  evenly  spread  across  the  spectrum. 

As  the  understanding  of  the  intelligibility  of  synthetic  speech  grew,  so,  too,  did 
interest  in  the  cognitive  demands  of  working  with  synthetic  speech.  Since  humans  find  it 
easier  to  understand  human  speech  than  synthetic  speech,  logic  suggests  that  its 
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interpretation  requires  the  use  of  more  cognitive  resources.  In  general,  increasing 
cognitive  load  is  not  necessarily  a  bad  thing.  In  fact,  one  of  the  goals  of  a  training  system 
is  to  increase  the  germane  load  in  order  to  produce  learning.  Due  to  limited  cognitive 
resources,  however,  learners  can  become  overwhelmed,  resulting  in  the  disengagement 
from  any  useful  learning.  For  this  reason,  it  is  important  to  reduce  extraneous  and  even 
intrinsic  load  to  the  greatest  extent  possible.  In  doing  so,  more  “space”  is  left  for  germane 
load  and  learning  that  is  more  efficient  is  possible. 

In  cases  where  the  duration  of  a  simulation  training  event  is  long  (greater  than  one 
to  two  hours)  or  requires  a  large  amount  of  adaptability,  the  use  of  synthetic  voice 
generation  may  be  more  advantageous  but  presents  the  possible  drawback  of  becoming 
extraneous  cognitive  load  (Hussain  &  Coleman,  2015).  Additional  cognitive  load  is  a 
specific  concern  when  training  ASCO  because,  as  discussed  above,  the  intrinsic  load  of 
the  job  is  already  restrictively  high.  Due  to  the  high  intrinsic  load  of  the  ASCO  task,  any 
training  system  should  seek  to  reduce  all  extraneous  loads  to  the  greatest  degree  possible. 
These  realities  present  a  challenging  problem.  The  use  of  synthetic  voice  generation  in 
such  a  training  simulation  is  more  cost  effective,  and  can  provide  more  adaptive  and 
relevant  training,  but  may  add  to  the  extraneous  load  and  potentially  reduce  the 
effectiveness  of  the  training.  However,  multiple  trials  of  dynamic  training  enabled  by  the 
use  of  synthetic  speech  generation  may  outweigh  the  potential  impacts  for  both  trainees 
and  instructors. 

Hussain  and  Coleman  (2015)  address  the  effect  of  synthetic  speech  on  mental 
workload.  They  point  out  that  the  use  of  synthetic  voice  has  several  advantages  and  may 
be  a  good  option  for  simulations.  They  emphasize  that  if  used  on  a  regular  basis,  trainees 
will  have  the  advantage  of  the  learning  the  effects  associated  with  synthetic  voice. 
Hussain  and  Coleman’s  also  pose  an  interesting  question:  could  listening  to  synthetic 
voice  improve  overall  listening  skills  by  requiring  the  trainee  to  focus  more  on  what  is 
being  said. 

Bowers,  Hussain  and  Procci  (2014)  sought  to  confirm  the  idea  that  using  synthetic 
voice  in  a  training  simulation  would  require  more  focus  and  result  in  deeper  learning.  To 
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do  so,  they  adapted  a  current  Navy  training  system  that  used  text  as  the  primary 
communication  medium  to  incorporate  synthetic  and  natural  voice.  The  participants,  119 
U.S.  Navy  recruits,  were  divided  into  two  groups  with  one  group  participating  in  the 
traditional,  text -based  training  approach  while  the  other  used  the  modified  approach. 
Upon  completion  of  the  training,  participants  in  each  group  listened  to  an  audio  clip  and 
were  asked  to  extract  important  information.  The  audio  clips  were  all  natural  voice,  but 
one  had  white  noise  added,  one  was  sped  up,  and  one  was  the  control.  The  intent  of  the 
difference  in  the  audio  clips  was  to  simulate  the  distortion  common  to  radio  transmission. 
They  found  no  evidence  that  using  synthetic  voice  in  the  training  improve  the  listener’s 
ability  to  extract  information  during  the  evaluation,  though  the  authors  do  suggest  that  a 
single  exposure  is  unlikely  to  result  learning. 

These  results  point  to  the  need  for  additional  research  in  the  area  of  transfer  of 
learning  effects.  It  is  unclear  if  issues  with  fluency  would  necessarily  apply  to  novices 
who  are  easily  overwhelmed  with  information.  However,  the  learning  effect  of  synthetic 
voice  may  still  prove  to  be  useful  if  it  can  be  shown  to  transfer  to  other  difficult  listening 
situations  such  as  military  radio  transmissions.  This  is  of  particular  interest  for  the 
training  of  ASCO  since  their  communications  are  dependent  on  the  use  of  radios 
transmitting  over  large  distances  in  less  than  ideal  environments. 

2.  Speech  Recognition 

Most  modern  speech  recognition  systems  use  a  combination  of  computer  learning, 
neural  networks,  and  statistical  reasoning  to  create  accurate  speech  recognition  systems 
(Chen  et.  al,  2006;  Gales,  1997;  Xiong  et.  al,  2016;  Zhu,  Stolcke,  Chen,  &  Morgan, 
2005).  Recent  breakthroughs  have  been  made  by  a  Microsoft  research  group  using 
convolutional  and  recurrent  neural  networks,  specifically,  a  Long-Short  Term  Memory 
(LSTM)  model  for  both  acoustic  and  language  modeling.  Xiong  et.al  (2016)  used  this 
combination  of  techniques  to  create  their  speech  recognition.  When  tested,  their  system 
achieved  better  performance  results  than  the  human  participants. 
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Like  most  speech  recognitions  systems,  the  goal  of  LSTM  is  to  model  natural 
speech  including  human  utterances,  self-corrections,  and  hesitations.  A  commonly  used 
data  set  for  testing  these  systems  was  created  by  the  National  Institute  of  Standards  and 
Technology.  The  most  difficult  to  understand  of  these  data  sets,  i.e.,  the  one  resulting  in 
the  most  incorrect  interpretations  by  speech  recognition  systems,  consisted  of  personal 
communication  over  the  telephone  (Cieri,  Miller  &  Walker,  2004).  These  data  sets  are 
particularly  difficult  due  to  the  poor  sound  propagation  over  telephones,  resulting  in  less 
accurate  acoustic  evaluation,  and  the  inconsistency  in  speech  inherent  to  a  natural 
conversation,  resulting  in  less  accurate  linguistic  evaluation. 

Even  with  systems  reaching  human  speech  recognition,  as  demonstrated  by  Xiong 
et.al  (2016),  a  system  that  focuses  on  natural  language  may  not  be  an  acceptable  option 
for  an  ASCO  individual  position  trainer.  As  stated  above,  most  of  these  systems  use 
statistical  analysis  to  decide  if  what  is  being  said  fits  what  is  expected  from  grammar  or 
training  sets  of  data.  ASCOs,  and  most  people  operating  on  military  communication 
systems,  do  not  use  normal  grammar.  Communications  over  these  systems  often  use 
acronyms,  sometimes  spelled  out  and  sometimes  said  as  word  (not  necessarily  an  English 
word).  This  domain  specific  communication  type  often  renders  most  systems  useless  due 
to  accuracy  issues.  However,  there  are  several  systems  that  allow  users  to  create  a 
grammar  specific  to  their  needs.  Several  of  these  options  were  considered  for  inclusion  in 
this  work.  The  outcome  of  this  consideration  will  be  further  explained  in  the  methods 
section. 
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IV.  METHODS 


This  section  examines  the  steps  taken  in  the  development  of  the  ASCO  individual 
position  training  prototype  simulation.  The  first  step  was  to  identify  training  requirements 
via  task  analysis.  The  next  step  was  to  identify  constraints  for  the  system  by  considering 
external  requirements,  such  as  usability,  availability,  and  development  time.  Using  this 
information,  we  built  a  conceptual  model  to  guide  product  selection  and  prototype 
development.  We  considered  several  commercial-off-the-shelf  products  were,  chose  the 
best  options  and  developed  a  prototype.  The  final  step  was  to  demonstrate  the  prototype 
to  subject  matter  experts  and  get  feedback  on  the  prototype  and  the  possible  addition  of 
such  a  system  into  ASCO  training. 

A.  TASK  ANALYSIS  METHOD 

The  task  analysis  was  conducted  using  a  modified  ADDIE  model.  The  first  step  in 
the  task  analysis  was  to  examine  the  DASC  training  and  readiness  manual  and  identify 
those  tasks  specific  to  ASCO  individual  position  training.  In  order  to  scope  the  project,  a 
single  task  was  chosen  and  broken  down  into  sub-tasks  and  associated  skills.  Out  of  these 
sub-tasks,  we  chose  to  focus  on  those  that  were  not  exclusive  to  specific  equipment  or 
environments.  For  example,  a  sub-task  of  “turn  on  the  radio”  may  vary  depending  on  the 
equipment  setup  being  used.  This  type  of  variable  task  was  not  considered  for  inclusion 
in  the  simulation  because  it  is  costly  to  simulate,  likely  to  become  obsolete  and  require 
costly  upgrades,  and  is  not  exclusive  to  ASCOs. 

Further  scoping  was  conducted  by  removing  tasks  related  to  document 
familiarization  such  as  “extract  information  from  the  air  tasking  order.”  Even  though 
these  types  of  tasks  are  important,  ASCOs  are  often  given  time  to  conduct  them  prior  to 
the  start  of  an  operation.  In  the  case  of  tasks  like  “conduct  airspace  familiarization,” 
ASCOs  have  several  days  to  study  the  airspace  prior  to  operations.  Because  these  tasks 
require  a  less  perishable  skill  set  and  relatively  large  amounts  of  time  are  allotted  for 
ASCOs  to  conduct  these  tasks,  they  are  not  included  in  the  prototype  simulation.  It  is 
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important  to  note  that  these  types  of  tasks  should  be  included  in  a  more  robust  simulation, 
and  it  would  likely  be  easy  to  add  them  with  the  creation  of  a  scenario  development  tool. 


B.  PROGRAMMING  APPROACH 

One  our  main  goals  when  deciding  on  a  programming  approach  was  modularity. 
The  initial  intent  was  to  create  models  of  non-ASCO  DASC  crew  positions  to  provide 
additional  stimuli  and  flexibility  to  the  system.  It  was  unclear  at  the  outset  of  this  project 
if  there  would  be  enough  time  to  create  these  models  and  logic  to  support  interaction  with 
the  user.  For  this  reason,  we  decided  to  modularize  the  models  to  support  extensibility. 

Even  though  the  development  of  a  user  interface  was  not  one  of  the  main  goals  of 
this  thesis  we  wanted  to  include  some  tools  to  support  scenario  modification.  For  this 
reason,  enumerations  of  specific  configuration  options  were  included  in  some  of  the 
models  as  publically  accessible  drop  down  menus. 

C.  ANALYSIS  OF  ALTERNATIVES 

An  analysis  of  alternatives  was  conducted  to  identify  available  simulation  and 
voice  recognition  /  synthetic  voice  production  software.  It  is  important  to  research  what 
options  are  available.  Oftentimes  conducting  an  analysis  of  alternatives  introduces 
options  that  were  not  previously  known,  and  identifies  the  best  design  options. 

For  this  thesis,  several  alternatives  were  identified  and  their  relative  merits  were 
considered.  Aspects  for  consideration  included  suitability,  usability,  availability, 
interoperability  and  transferability.  A  suitable  option  meets  required  performance 
standards.  A  usable  option  require  minimal  front  end  training.  Available  options  do  not 
require  special  permission  or  certification  and  are  not  restrictively  expensive. 
Interoperable  systems  are  able  to  integrate  with  other  selected  systems.  A  transferable 
system  is  one  that  can  transition  to  military  use  or  is  already  certified  for  military  use. 

D.  SUBJECT  MATTER  EXPERT  EVALUATION 

The  final  step  was  to  show  the  ASCO  training  system  prototype  to  subject  matter 
experts  (SMEs)  and  elicit  feedback  on  the  system’s  perceived  utility.  Each  SME  was 
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provided  with  a  questionnaire  prior  to  a  demonstration  of  the  simulation.  This  first 
portion  of  the  questionnaire  focused  on  current  ASCO  individual  position  training.  The 
second  part  of  the  questionnaire  was  distributed  after  the  demonstration  and  focused  on 
the  possible  effects  such  a  simulation  could  have  on  training,  what  positives  and 
negatives  the  system  offers,  and  what  additions  would  improve  the  utility  of  the 
simulation.  The  results  of  the  project  and  the  feedback  from  the  demonstration  are 
discussed  in  the  results  section  of  this  thesis. 
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V.  RESULTS 


This  section  describes  the  functionality  achieved  in  the  prototype  simulation,  the 
capabilities  of  the  resultant  grammar  and  the  input  provided  by  subject  matter  experts. 

A.  TASK  ANALYSIS  RESULTS 

The  overarching  task  identified  for  use  was  AIRS- 1152:  Perform  as  a  Tactical  Air 
Director.  Figure  1  is  an  excerpt  from  the  DASC  Training  and  Readiness  Manual 
outlining  the  subtasks  associated  with  AIRS-1152.  Highlighted  are  the  subtask  and 
mission  sets  that  were  further  broken  down  and  analyzed  for  inclusion  in  the  prototype 
simulation.  Outlined  in  blue  are  those  tasks  analyzed  and  included  in  the  prototype 
simulation. 


AIRS-1152 *  B 


E 


G 


Goal .  Perform  as  a  Tactical  Air  Director 

Requirement .  In  a  DASC,  perform  the  following: 

1.  Parse  the  ATO  to  create  accurate  soft  frags. 

-  -Ay-  P.c.-.tc-  —  -  - 

3.  De-conflict  aircraft  with  surface  to  air  threats,  other 
aircraft,  and  combined  arms  fires. 

4.  Use  proper  terminology  associated  with  procedural  control. 

5.  Use  the  SALT  brief  to  RIO  F/W  aircraft. 

6.  Accurately  control  common  F/W  missions: 

a.  CAS 

b .  XCAS 

“ -  c.  FA(i  (A)  - - - — - - 

d.  Reconnaissance 

e.  Refueling 

f .  SCR 

g.  UAS 

h.  Downed  A/C 

1.  Pass  and  receive  air  support  requests  to  and  from  F/W 
aircraft . 

8.  Give  a  crew  brief. 

Performance  Standard.  Without  the  aid  of  reference,  pass  a  test 
with  a  minimum  score  of  80%  and  apply  knowledge  gained  in 
practical  application  using  the  DASC. 

Reference . 

1.  MCWP  3-25.5 

2.  DASC  Pocket  Checklist 

Figure  1.  AIRS-1 152  Extracted  from  Training  and  Readiness  Manual. 

Source  DoN  USMC  (2013). 
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These  sub-tasks  were  further  decomposed  in  to  step-by-step  processes.  This 
helped  us  to  identify  how  the  simulation  should  run,  the  visual  and  auditory  information 
should  be  presented  to  the  ASCO,  and  what  inputs  the  ASCO  should  be  able  to  provide  to 
the  system.  The  following  is  a  list  of  the  displays,  and  inputs  and  outputs  identified  for 
inclusion. 

Displays 

•  Airspace  map  (see  Figure  2) 

•  Radio  in/out  board  (see  Figure  3) 

•  Chat  window  (see  Figure  4) 

Output 

•  Audio  of  pilot  communication 

•  Visual  representation  of  aircraft  (to  be  used  as  scaffolding) 

•  Visual  information  describing  airspace  status 

Input 

•  Verbal  communication  of  instructions  to  pilots 

•  Type  in  dissemination  of  mission  information 

The  fixed  wing  (F/W)  mission  type  selected  for  inclusion  was  close  air  support 
(CAS).  The  following  is  a  chronological  list  of  events  during  an  inbound  CAS  mission. 

1.  Aircraft  check-in 

2.  Routing 

3.  Safety  of  flight  brief 

4.  Air  tasking  order  confirmation 

5.  Air  defense  warning  condition  weapons  control  status  brief 

6.  Follow-on  instructions 

7.  Handover  to  terminal  controller 

After  a  pilot  switches  to  terminal  control,  they  are  no  longer  in  communication 
with  the  DASC.  Once  the  CAS  mission  is  complete,  the  pilot  contacts  the  DASC  for 
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return-to-base  routing.  The  following  list  is  the  chronological  steps  of  the  retum-to-base 
at  the  conclusion  of  a  CAS  mission. 

1 .  Aircraft  check-in 

2.  Confirm  fuel  and  ordnance  status 

3.  Routing 

4.  Safety  of  flight  brief. 

5.  In  flight  report 

6.  Air  defense  warning  condition  weapons  control  status  brief 

7.  Follow-on  instructions 

8.  Handover  to  air  traffic  control  (ATC) 

Once  the  task  analysis  was  complete  we  began  to  consider  what  product  would 
help  to  support  the  development  of  the  system.  To  further  guide  the  product  decision  we 
identified  several  constraints  in  the  analysis  of  alternatives. 


Figure  2.  Map  Screen  Capture  from  Prototype  Simulation 
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Figure  3.  RIO  Board  Screen  Capture  from  Prototype  Simulation 


Figure  4.  Chat  Window  Screen  Capture  from  Prototype  Simulation 
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B.  ANALYSIS  OF  ALTERNATIVES  RESULTS 


Using  the  information  gathered  during  the  task  analysis,  we  identified  the 
components  of  a  DASC  that  the  prototype  simulation  would  encompass.  Mockups  were 
made  of  each  of  these  components  and  the  flow  of  information  from  one  component  to 
the  next  was  identified. 

1.  Constraints 

In  order  to  improve  the  availability  and  reduce  the  cost  of  simulations,  several 
constraints  were  applied.  First,  the  prototype  main  hard  ware  consists  of  a  single  laptop. 
This  makes  the  system  easy  to  transport  and  reduces  the  amount  of  space  to  house  the 
system.  Ultimately,  this  small  foot  print  makes  the  system  more  affordable  and  accessible 
if  adopted  and  fielded.  The  only  requirement  for  the  hardware  was  that  it  works  with  the 
selected  software. 

The  requirements  to  access  a  military  network  are  very  stringent  and  take  a 
considerable  amount  of  time  and  effort  to  complete.  In  order  to  facilitate  the  possibility  of 
testing  the  prototype  at  a  MASS,  we  decided  that  the  system  should  run  without  an 
Internet  connection.  The  intent  was  to  improve  the  ease  of  testing  and  make  the  system 
easier  to  deploy. 

As  previously  mentioned,  one  of  our  main  goals  is  to  improve  the  availability  of 
training  for  ASCO.  One  way  to  accomplish  this  is  to  reduce  the  personnel  requirement 
associated  with  training.  To  this  end,  another  requirement  is  for  the  prototype  simulation 
to  be  operable  by  a  single  trainee. 

2.  Hardware 

The  selected  hardware  includes  an  Alienware  laptop  and  a  Plantronics  vivox 
headset  with  incorporated  microphone  and  adjustable  volume  control.  The  laptop 
includes  an  Intel  i7  processor  with  a  2.80GHz  CPU,  16  GB  of  RAM,  a  64-bit  operating 
system  and  Windows  8.1  Pro.  The  choice  of  the  laptop  was  the  result  of  a  software  quirk. 
The  Lexix  uses  the  computers  serial  number  to  create  a  computer  specific  identification 
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number.  The  serial  number  on  the  original  computer  was  masked  so  one  of  the  school’s 
computers  was  used  instead. 

The  headset  is  not  a  requirement  but  improves  the  usability  of  the  system  in  a 
noisy  environment  and  is  similar  to  the  headsets  worn  by  ASCOs.  It  would  be 
advantageous  to  procure  a  headset  that  had  an  incorporated  push  to  talk  button.  The 
current  system  requires  the  user  to  use  a  mouse  to  select  the  push  to  talk  button  on  the 
Lexix  user  interface.  By  doing  this,  the  user  is  deselecting  the  simulation  to  select  the 
voice  recognition  system.  To  use  the  simulation’s  keyboard  interactions,  the  user  has  to 
reselect  the  simulation  using  the  mouse.  This  is  a  slight  inconvenience  during  the 
demonstration  of  a  prototype  but  would  significantly  affect  the  usability  of  a  real 
simulation. 


3.  Simulation  Software 

Several  products  were  considered  to  be  used  as  the  backbone  of  the  prototype 
simulation  including  Sculpt,  Flames,  and  the  Unity  game  engine.  The  advantage  of  Sculpt 
and  Flames  is  that  they  are  designed  to  work  with  the  common  aviation  command  and 
control  system  (CAC2S)  by  generating  data  link  information  as  an  output.  The  CAC2S  is 
the  command  and  control  system  used  by  the  MACCS. 

Sculpt  is  a  proprietary  scenario  generating  system  developed  by  Raytheon 
Solipsys.  It  uses  graphical  user  interfaces  to  simplify  the  task  of  creating  scenarios. 
Operators  are  able  to  preplan  routes  for  aircraft  and  instantly  update  them  during 
playback.  It  also  gives  the  operator  the  ability  to  dynamically  introduce  new  objects  in 
order  to  change  or  increase  the  complexity  of  the  scenario  on  the  fly  (Raytheon,  n.d.). 
Because  Sculpt  is  a  closed  system,  we  would  be  restricted  to  interacting  with  the 
simulation  in  limited  ways.  This  would  preclude  the  use  of  voice  interaction  without 
addition  of  a  human  interpreter.  Since  voice  communication  is  a  key  skill  for  ASCO, 
Sculpt  was  not  used  for  the  prototype. 

FLAMES  is  a  simulation  training  solution,  produced  by  the  Temion  Corporation. 
Unlike  Sculpt,  FLAMES  is  not  a  single  purpose  simulation,  but  a  framework  for 
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developing  simulations.  This  gives  FLAMES  an  inherent  flexibility  but  also  implies  a 
certain  level  of  programming  overhead  (Ternion,  n.d.).  To  use  FLAMES,  we  would  have 
to  learn  how  to  program  within  that  framework.  We  would  be  able  to  incorporate  voice 
communication  using  FLAMES,  but  there  is  no  integrated  voice  recognition  system.  We 
would  have  to  build  a  bridge  to  get  FLAMES  to  work  with  whichever  voice  recognition 
system  we  chose.  For  these  reasons,  FLAMES  was  not  used  for  the  prototype  . 

The  Unity  game  engine  is  a  freely  available  game  engine.  Like  FLAMES,  it  is 
flexible  in  that  we  can  program  it  to  do  almost  anything  we  require.  Furthermore,  several 
voice  recognition  packages  have  Unity  plugins.  In  addition  to  these  positives,  there  are 
several  Unity  experts  in  the  MOVES  Institute  available  for  consultation  and  students 
receive  an  introduction  to  Unity  as  a  part  of  the  curriculum.  For  these  reasons  Unity  was 
chosen  as  the  backbone  to  the  prototype. 

4.  Voice  Recognition  Software 

During  the  conduct  of  the  task  analysis  it  became  obvious  that  two-way  voice 
communication  is  a  key  aspect  of  ASCO  tasks.  As  mentioned  before,  all  current  training 
solutions  achieve  this  with  the  use  of  role-players.  As  outlined  in  the  constraints,  it  was 
our  intent  to  create  a  system  that  a  single  trainee  could  use  without  such  external  support. 
The  use  of  voice  recognition  and  synthetic  voice  production  was  identified  as  the  best 
option  to  produce  a  system  that  met  that  requirement. 

Several  voice  recognition  systems  were  considered  for  inclusion  in  the  prototype 
training  simulation,  but  the  majority  had  difficulty  meeting  our  requirements.  Most 
commercial  off-the-shelf  (COTS)  speech  recognition  systems  are  made  to  identify  normal 
speech  patterns,  i.e.,  full  sentences  using  common  words  spoken  at  the  usual  speed  of 
speech.  The  type  of  communications  passed  over  the  radio  between  the  DASC  and  pilots 
does  not  constitute  normal  speech  patterns.  For  example,  “Are  you  up  as  fragged?”  is  a 
common  phrase  used  to  confirm  that  there  has  been  no  change  to  an  aircraft’s 
configuration  since  the  publication  of  its  mission  data.  The  word  “fragged”  is  not  part  of 
normal  English  but  is  a  common  military  term.  In  addition  to  not  recognizing  military 
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radio  speech,  the  speed  of  speech  used  during  radio  communication  overwhelmed  most 
systems.  Asking  an  ASCO  to  slow  down  their  rate  of  speech  so  that  a  COTS  system 
could  properly  parse  the  utterances  would  likely  produce  negative  training.  ASCOs  are 
often  talking  to  several  different  aircraft,  are  required  to  speak  in  ten  second  bursts  for 
emission  control,  and  often  have  pass  as  much  information  as  possible  between  static 
bouts  of  static  interference.  It  was  necessary  to  find  a  system  able  to  handle  rapid  speech 
using  a  specialized  lexicon.  Table  2  outlines  the  voice  recognition  systems  considered  for 
inclusion. 


Table  2.  Voice  Recognition  Systems 


System  name 

Unity  plug-in 

Supports 

programmable 

grammar 

Apparent 

accuracy 

Requires 

Internet 

connection 

Watson 

(streaming) 

Yes 

No 

Poor 

Yes 

Watson  (non¬ 
streaming) 

Yes 

No 

Fair 

No 

Wit.ai 

Yes 

No 

Fair 

No 

Nuance  Dragon 
Speech 

No 

Yes 

Good 

No 

Windows  Dictation 

Yes 

Yes 

Poor* 

No 

Adacel  Lexix 

Yes 

Yes 

Good 

No 

*Without  programming  grammar 


Of  all  the  candidate  systems,  Adacel’s  Lexix  was  the  only  software  that  delivered 
the  required  capabilities.  Adacel  specializes  in  ATC  simulations.  Even  though  the  DASC 
controls  airspace  in  a  very  different  way  than  the  ATC,  the  voice  recognition 
requirements  are  similar,  such  as  the  rate  of  speech  and  some  of  the  base  grammar.  Much 
of  the  DASC  specific  communication  had  to  be  added  but,  Lexix  handled  the  quick 
speech  of  controllers  and  pilots.  Adacel  provided  programmers  with  access  to  the 
dictionaries  used  to  support  the  voice  recognition  software,  which  enabled  us  to  create 
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words  and  word  combinations  not  found  in  normal  language,  allowing  us  to  add  the 
required  components  of  the  DASCs  grammar.  In  addition  to  providing  voice  recognition, 
Lexix  also  provides  synthetic  voice  production,  which  was  used  in  our  prototype 
simulation. 

C.  PROTOTYPE  DEVELOPMENT 

1.  Simulation  Development 

Development  of  the  main  simulation  began  with  the  creation  of  mockups  of  the 
three  main  DASC  displays.  The  main  components  of  each  display  and  the  required 
functionality  identified  during  the  task  analysis  were  used  to  create  the  mockups.  These 
mockups  where  imported  into  Unity  and  served  as  a  guide  to  build  the  simulated 
displays.  Since  all  DASC  displays  are  two-dimensional,  we  decide  to  use  two- 
dimensional  displays  in  the  simulation.  This  also  reduced  development  time  and  saved  on 
the  amount  of  processing  power  required  to  run  the  simulation. 

After  the  displays  were  created,  models  of  representative  aircraft  were  developed. 
Aircraft  were  give  attributes  to  represent  flight  information,  mission  information,  and 
configuration  information.  For  example,  an  aircraft  can  be  an  F-18  on  a  CAS  mission 
scheduled  for  0830.  It  will  initialize  at  10,000  feet,  at  a  given  control  point,  and  be  armed 
with  two  AIM-9s.  To  facilitate  voice  communications,  certain  attributes  were  given 
nomenclature  and  one  to  three  spoken  names.  The  AIM-9,  for  example,  has  the  given 
nomenclature  AIM-9,  the  spoken  name  “aim  nine,”  and  the  spoken  name  “Sidewinder.” 
Instantiated  aircraft  are  capable  of  moving  around  the  airspace  and  interacting  with 
certain  objects  via  collision  detection.  For  example,  an  aircraft  will  recognize  when  it  is 
flown  into  a  threat  ring. 

Certain  aspects,  such  as  an  ordinance  list,  were  included  as  an  enumeration.  This 
allowed  us  to  create  drop  down  menus  to  easily  adjust  attributes  of  our  models.  Certain 
attributes  allow  for  the  input  of  a  string  element,  which  can  be  used  by  the  synthetic  voice 
production  system  to  provide  spoken  information  to  the  user.  An  example  of  this  is 
exceptions.  If  an  aircraft  is  airborne  with  a  different  configuration  that  is  published  prior 
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to  the  mission,  they  are  considered  to  be  “up  with  exceptions.”  It  is  the  responsibility  of 
the  ASCO  to  confirm  the  aircraft  is  configured  as  published.  If  it  is  not,  the  ASCO  asks 
the  pilot  to  “state  exceptions”  and  the  pilot  reports  all  of  the  exceptions,  i.e.,  “I  am  a 
flight  of  one  vice  two  due  to  maintenance.” 

A  mediator  class  was  created  to  handle  the  information  flowing  between  all  of  the 
displays,  models,  and  voice  recognition  system.  This  class  keeps  all  of  the  main 
components  separate  and  unaware  of  each  other.  This  approach  provides  modularization 
to  improve  extensibility. 

The  final  step  was  to  connect  the  voice  recognition  and  synthetic  voice  production 
and  program  the  mission  logic  into  the  system.  Before  we  could  do  this  we  had  to  adapt 
the  Lexix  grammar  to  handle  DASC  communications. 

2.  Grammar  Development 

Creating  the  grammar  for  the  DASC  prototype  simulation  was  the  most  time 
consuming  step  in  the  development  process.  Lexix  follows  the  W3C  speech  recognition 
grammar  specifications  version  1.0.  The  language  is  constructed  on  several  levels.  The 
base  dictionary  contains  every  allowed  word  and  its  phonetic  pronunciation.  As 
mentioned  above,  Adacel  provided  access  to  the  base  dictionary,  which  enabled  the 
addition  of  uncommon  words;  however,  we  did  not  have  to  add  any  words  to  the  base 
dictionary. 

The  next  level  parameterizes  commonly  used  words  and  word  combinations.  This 
section  is  largely  made  up  of  numbers  and  the  phonetic  alphabet.  One  of  the  main 
additions  at  this  stage  was  time  combinations  and  the  military  grid  reference  system.  It 
was  necessary  to  account  for  the  fact  that  there  might  be  multiple  acceptable  ways  to  say 
the  same  thing  and  the  system  needed  to  be  able  to  recognize  all  of  them.  For  example,  it 
is  common  in  the  military  for  some  to  refer  to  a  time  as  “zero  five  thirty,”  or  “oh  six 
hundred,”  or  “zero  seven  one  five.”  By  programming  the  grammar  to  recognize  time  as  a 
parameter,  we  can  now  tell  the  system  to  recognize  a  sentence  such  as  “the  time  is  now 
$time”  (where  “$time”  is  any  acceptable  way  to  state  a  time  as  defined  by  the  grammar). 
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The  next  level  deals  with  dynamic  grammar  parameters.  These  parameters  are 
specific  to  a  simulation  or  a  scenario.  They  are  used  to  connect  part  of  the  ASCO’s 
utterance  to  a  specific  object  in  the  simulation.  For  example,  one  of  the  aircraft  in  our 
simulation  has  the  call  sing  “Combat  22.”  The  dynamic  grammar  allows  you  to  input  key 
value  pairs  to  be  recognized  as  a  parameter.  “Combat  22”  is  recognized  as  a  call  sign  and 
the  key  CT22  is  returned  when  the  call  sign  is  spoken  in  a  recognized  sentence  such  as 
“go  ahead  $Callsign.”  The  simulation  uses  the  key  to  determine  how  to  handle  the  results 
of  parsing  the  utterance.  In  this  case,  if  the  ASCO  says,  “Go  ahead  Combat22.,”  the 
simulation  passes  the  message  (“Go  ahead”)  to  the  object  with  the  key  CT22,  and  that 
object  creates  and  delivers  a  response  to  the  message. 

The  next  level  identifies  available  parameters  to  the  system  and  enables  increased 
complexity  through  the  combination  and  concatenation  of  parameters.  An  example  of  this 
is  building  a  route  from  a  series  of  control  points.  The  control  points  are  added  on  the 
previous  level  as  a  set  of  dynamic  grammar  parameters.  This  set  consists  of  individual 
control  points  described  with  key  value  pairs —  for  example,  SH  =  Sarah,  KM  =  Kim,  EY 
=  Emily.  On  this  new  level  you  can  create  more  complex  parameters  by  specifying 
varying  combinations  of  different  or  similar  parameters.  Our  simulation  allows  for  the 
concatenation  of  up  to  five  control  point  to  define  a  route.  If  a  user  says  “Proceed  Sarah, 
Emily,  Kim”  or  “Proceed,  Kim,  Tina,  Sarah,  Heather,”  the  system  will  recognize  it  and 
return  an  appropriate  key  (SHEYKM  or  KMTASHHR).  The  key  is  parsed  and  used  to 
build  a  route  for  an  aircraft,  which,  in  turn,  is  used  to  guide  the  aircraft’s  movements. 

The  next  level  is  the  instruction  level.  At  this  level  all  of  the  parameters  that  will 
be  used  in  a  given  command  are  identified.  For  example,  when  passing  a  safety  of  flight 
brief  for  a  firing  artillery  battery  several  parameters  are  required,  such  as  the  point  of 
origin,  the  point  of  impact,  the  maximum  ordinate,  and  the  time  of  the  fire  mission.  The 
instruction  lists  all  of  these  parameters.  It  is  also  possible  create  an  instruction  with  no 
parameters,  for  instance,  “break”  is  a  common  radio  phrase  that  means  you  are  breaking 
the  connection  for  a  second  but  still  have  more  information  to  pass.  No  parameters  are 
required  for  this  type  of  transmission. 


35 


The  final  level  is  the  command  level.  Commands  are  the  actual  phrases  the  system 
will  recognize.  This  include  the  parameters  and  all  connecting  words.  Using  the  W3C 
speech  recognition  grammar  specifications  version  1.0  a  single  line  can  be  used  to 
construct  several  phrases.  Optional  words  are  set  off  by  square  brackets,  e.g.,  [and],  while 
places  where  multiple  words  might  work  are  set  off  by  parentheses  and  separated  within 
the  parentheses  by  a  vertical  line,  e.g.,  (a  I  an).  For  example,  if  the  required  sentence  is- 
“[be  advised]  (there  is  I  you  have)  an  $Aircraft”  and  $Aircraft  is  defined  in  the  dynamic 
parameters  as  “F/A-18”  or  “AH-1W,”  all  of  the  following  are  acceptable  phrases. 

•  “Be  advised  there  is  an  F/A-18” 

•  “Be  advised  you  have  an  F/A-18” 

•  “There  is  an  F/A-18” 

•  “You  have  an  F/A-18” 

•  “Be  advised  there  is  an  AH-1W” 

•  “Be  advised  you  have  an  AH- 1W” 

•  “There  is  an  AH-1W” 

•  “You  have  an  AH-  1W” 

Lexix  is  configured  to  recognize  up  to  four  commands  in  a  single  transmission. 
Because  of  this  limitation,  it  is  important  to  consider  how  parameters  and  instructions  are 
constructed.  The  communication  of  routes  is  an  excellent  example  of  this.  As  mentioned 
above,  we  created  a  parameter  that  concatenates  several  control  points  into  a  single  route. 
This  approach,  versus  allowing  each  control  point  to  constitute  a  single  command, 
allowed  us  to  account  for  complex  communication  string  without  over-whelming  the 
system.  For  example,  the  phrase  “climb  to  5000  feet  (1st  command),  proceed  Kim  (2nd 
command),  Emily  (3rd  command),  Sarah  (4th  command)”  would  utilize  all  available 
system  resources  for  a  single  transmission.  This  approach  would  cause  an  error  unless  the 
user  included  a  transmission  break.  Based  on  the  task  analysis,  it  was  clear  that 
communication  like  this  would  not  be  natural  for  ASCO  operations  and  would  introduce 
the  potential  for  negative  training.  For  the  mission  set  to  be  appropriately  represented  in 
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our  prototype  simulation,  we  developed  parameters  capable  of  mitigating  this  problem. 
This  solution,  however,  does  not  cover  all  potential  problems.  Other  mission  sets,  such  as 
the  passing  of  a  “nine  line”  for  casualty  evacuation,  which  has  nine  pieces  of  information 
to  pass  could  prove  a  difficult  problem. 

Because  there  is  no  standardized  phrase  book  governing  DASC  communications, 
grammar  was  developed  hand-in-hand  with  the  mission  logic  revealed  in  the  task 
analysis.  After  we  completed  one  step  of  the  mission  we  went  on  to  the  next.  The 
grammar  for  the  new  mission  step  was  diagramed  and  programmed  and  then  the 
simulation  was  updated  to  recognize  and  respond  to  the  new  command.  As  the  system 
was  tested  common  phrases  that  are  not  specific  to  mission  sets,  such  as  “stand  by”  and 
“say  again”  were  identified  and  added. 

Lexix  is  able  to  recognize  only  the  specific  commands  that  it  has  been 
programmed  to  recognize.  This  improves  accuracy  by  reducing  search  space.  Initially  we 
did  not  remove  any  of  the  pre-programmed  ATC  specific  grammar.  As  the  size  of  the 
DASC  grammar  increased,  the  accuracy  began  to  decrease.  In  order  to  improve  accuracy 
of  the  system  we  removed  all  of  the  ATC  specific  grammar.  This  had  the  desired  effect 
and  did  improve  accuracy. 

Another  problem  we  ran  in  to  was  the  confusion  of  similar  words.  In  our  case,  this 
was  isolated  to  the  control  points  -  specifically  Kim,  Ken,  and  Ben.  We  resolved  this  by 
removing  similar  sounding  control  point.  It  is  important  to  note  that  the  use  of  similar 
sounding  control  points  is  bad  practice  for  airspace  design  as  they  can  easily  be  confused 
over  the  radio  during  actual  operations,  so  this  is  not  a  limitation  of  the  system. 

3.  The  Final  Product 

The  prototype  system  that  resulted  from  this  work  is  capable  of  simulating  the 
basic  tasks  of  an  ASCO  controlling  a  CAS  mission.  Pilot  -  ASCO  interactions  are 
conducted  via  voice  communication.  The  system  responds  to  ASCO  commands  in 
accordance  with  provided  mission  logic.  For  example  if  the  ASCO  says  “climb  to  10,000 
feet  and  proceed  to  Sarah,  contact  this  station  Sarah”  the  aircraft  model  climbs  at  a 
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designated  rate  until  it  reaches  10,000  feet  and  moves  at  a  designated  speed  to  control 
point  Sarah.  Once  the  aircraft  model  reaches  control  point  Sarah  it  reports  that  it  is 
“established  Sarah.”  Each  of  the  mission  steps  identified  during  the  task  analysis  have 
been  included  in  the  mission  logic  and  models  behave  as  expected  for  each  step. 

The  map  (Figure  2)  provides  visual  feedback  to  the  ASCO.  It  acts  as  a  visual 
representation  of  the  airspace.  The  map  is  updatable,  but  currently  the  only  portion  that 
updates  automatically  is  the  clock  and  the  visual  representation  of  the  aircraft  as  it 
traverses  the  airspace.  In  addition  to  the  airspace,  pertinent  airspace  information  is 
depicted  on  the  right  hand  side  of  the  map  and  is  updatable.  This  provides  the  ASCO  the 
ability  to  visually  reference  changeable  airspace  information  to  incorporate  in  to  air  space 
briefs. 

The  RIO  board  (Figure  3)  exhibits  the  desired  behavior.  Upon  initiation,  the 
simulation  instantiates  all  aircraft  missions.  The  relevant  information  is  gathered  from  the 
aircraft  models  and  populates  the  “upcoming  missions”  section  of  the  RIO  board.  Once 
the  aircraft  checks  in  with  the  ASCO  the  mission  is  moved  to  the  “airborne”  section. 
Throughout  the  mission  the  RIO  board  updates  automatically  with  the  aircraft’s  location 
and  altitude  information.  After  an  aircraft  has  been  turned  over  to  ATC,  the  associated 
mission  block  is  moved  to  the  “completed  mission  section. 

The  chat  windows  (Figure  4)  provide  the  opportunity  to  practice  passing  mission 
information  to  internal  and  external  agencies.  Currently,  the  ability  to  input  information  is 
the  only  function  used  in  the  simulation.  However,  the  ability  to  automatically  update  the 
chat  windows  is  available  and  would  be  used  to  provide  information  to  ASCOs  from 
internal  and  external  agencies. 

The  voice  recognition  system  accurately  recognizes  relevant  ASCO  commands 
and  most  commonly  used  phrases.  Thirty-one  commands  resulting  in  347  supported 
phrases  have  been  programmed  in  to  the  system.  The  commands  include: 

•  Acknowledge  •  ATO  confirmation 

•  Altitude  •  Barometer 
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•  Break 

•  Call  sign  (initiate  contact) 

•  Climb  to  altitude 

•  Climb  to  flight  level 

•  Air  defense  warning 
condition  weapons 
control  status  explanation 

•  Descend  to  altitude 

•  Descend  to  flight  level 

•  Flight  level  change 

•  Follow  on  instructions 

•  Control  point  reporting 

•  Fuel  and  ordinance  status 

•  How  copy 

•  In  flight  report 

•  State  palmate  call  sign 


•  State  present  position  and 
altitude 

•  Proceed  on  control  point 
route 

•  Roger  up 

•  Enemy  safety  of  flight 

•  Say  again 

•  State  exceptions 

•  Artillery  safety  of  flight 

•  Friendly  traffic  safety  of 
flight 

•  Stand  by 

•  Stand  by  for  grid 

•  Switch  to  terminal  control 

•  Report  time 

•  Warning  condition 


These  commands  are  sufficient  to  support  a  CAS  mission  from  beginning  to  end. 
The  grammar  will  have  to  be  expanded  to  support  further  mission  sets  but,  due  to  the 
overlapping  requirements  of  most  mission  sets,  much  of  the  grammar  developed  to 
support  our  prototype  simulation  would  support  other  missions. 

D.  SUBJECT  MATTER  EXPERT  RESULTS 

To  gage  the  efficacy  of  the  prototype  simulation  SMEs  were  asked  to  complete  a 
questionnaire.  The  first  half  of  the  questionnaire  focused  on  current  effectiveness  and 
frequency  of  current  ASCO  individual  position  training.  Table  3  shows  the  questions 
asked  and  relevant  extracts  from  their  answers.  Some  modifications  have  been  made  to 
improve  readability.  The  complete  questionnaire  and  answers  can  be  found  in  appendix  A 
and  C. 


39 


Table  3.  SME  Questions  and  Response  Pre-Demonstration 


Question 

SME  1 

SME  2 

1.  Dose  a  gap  exists  for 
air  support  control 
officer’s  individual 
position  training?  If  so 
describe  the  gap. 

“Once  out  of  MOS  schools,  ‘Individual’  position 
training  does  not  truly  exist.  A  significant  number 

2000  level  T&R  events  (individual  training  events), 
actually  require  a  crew  and  a  white  cell  to  execute. 
Because  . . .  many  of  the  2000  level  training  events 
require  a  crew  and  a  white  cell  to  execute,  training 
opportunities  . . .  arise  when  DASC  Drills  are 
scheduled  not  based  on  the  individual’s  training 
requirement  for  progression.  This  limits  the  number  of 
reps  that  an  individual  can  get  in  a  given  year,  limiting 
their  proficiency  as  a  result.” 

“Yes,  right  now  the  only  individual  training  occurs  at 
MOS  school.  There  is  very  minimal  training  that 
focuses  on  an  individual  controller.  The  most 
individual  training  a  controller  gets  is  practicing  Radio 
in  and  out  (RIO)  with  other  officers.  “ 

2.  What,  if  any, 
individual  position 
training  is  conducted 
prior  to  a  DASC  drills? 

“There  are  very  few  T&R  events  that  are  conducted 
prior  to  DASC  Drills.  This  is  partly  due  to  the  factors 
mentioned  above,  but  also  [due]  to  the  high  op  tempo 
of  the  Squadrons.  DASC  Drills  are  typically  crammed 
in  between  exercises  that  squadron  is  supporting,  and 
because  most  2000  and  3000  T&R  events  inherently 
require  a  crew,  a  whitecell,  and  a  qualified  evaluator  it 
rarely  occurs” 

“Like  I  stated  above,  the  only  individual  position 
training  is  practicing  RIOs  with  another  officer  (most 
of  the  time  this  is  limited).” 

3.  What,  if  any, 
individual  position 
training  is  conducted 
prior  to  live  exercises 
or  operations? 

“Prior  to  combat  operations,  the  squadrons  do  a  pretty 
good  job  of  running  DASC  drills  using  a  scenario  that 
mimics  the  theater  of  operations.  But  this  is  rarely 
“individual”  for  the  reasons  mentioned  above.” 

“Very  minimal,  if  there  is  time  a  more  senior  officer 
might  practice  and  mentor  a  junior  officer.  Most  of  the 
time  more  senior  officers  are  busy  planning  and 
prepping  for  exercises  or  operations,  so  very  little  time 
if  any  is  spent  on  individual  training.  Most  of  the 
training  is  conducted  at  the  crew  level,  making  sure 
the  crew  is  ready  to  operate  as  a  whole.  “ 
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Question 

SME  1 

SME  2 

4.  a.  How  often  (in 
months)  does  this 
training  occur? 

“We  tried  to  do  a  DASC  drill  once  a  quarter  but  this 
was  driven  by  the  squadron’s  op  tempo.” 

“Every  4  months. . .  at  best!” 

4.  b.  How  is  this 
training  evaluated 
(sat/unsat,  grades, 
etc.)? 

“Performance  evaluation  is  captured  on  a  MACCS 
training  form,  0-4  scale  with  zero  being  unsat  and  4 
representing  mastery.  A  student  must  score  at  least  a 

3  (proficient)  in  order  to  receive  credit  for  the  T&R 
event.  However,  this  is  purely  a  qualitative 
evaluation  by  evaluator.  Rarely  are  an  actually 
metrics  included  in  the  performance  evaluation.  “ 

“None” 

4.  c.  List  the  types 
feedback  which  is 
provided  to  the 
trainee  (real-time, 
after  action  review, 
none,  etc.)? 

“After  every  crew  executes  the  drill,  there  is  a  quick 
hot  wash  by  the  evaluators.  This  is  purely  qualitative. 
No  actual  metrics  are  collected  and  used  to  measure 
performance.  Additionally,  each  evaluator  is  supposed 
to  debrief  the  individual  they  were  evaluating  on  their 
performance  in  greater  detail  providing  areas  for 
improvement.” 

“None” 

4.  d.  Are  records  of 
this  training 
maintained? 

“Yes  on  the  MACCS  Training  Forms” 

“No” 

4.  e.  How  are  these 
records  used  to 
inform  future  training 
of  individual  AS  CO 

“WTTP  is  supposed  to  look  at  what  training  events  an 
individual  has  completed  and  what  training  events 
they  need  to  complete  in  order  to  get  qualified,  and 
then  build  drills  to  meet  the  requirements.  As  1 
mentioned  previously,  very  few  WTTP  officers  have 
the  mental  agility  to  make  this  happen.  Typically  a 
scenario  is  built,  individuals  sit  position  on  a  crew, 

“N/A” 
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Question 

SME  1 

SME  2 

and  evaluators  must  create  the  training  in  real  time  in 
order  for  the  individual  to  get  the  X.” 

4.  f.  How  are  these 
records  used  to 
inform  the  squadron 
training  plan? 

“The  WTTP  officer  was  tasked  with  identifying  who 
needed  to  get  which  qualification  in  order  to  make  all 
of  our  crews  qualified.  Once  that  was  accomplished, 
he  had  to  identify  what  training  requirements  each 
individual  needed  in  order  to  receive  position 
qualification.  Based  on  the  requirements  of  the 
individuals,  WTTP  was  tasked  with  coming  up  with 
individualized  training  plans  and  a  squadron  training 
plan  that  supported  satisfaction  of  each  individual’s 
requirement.  This  was  too  much  work  for  my  first 
WTTP  officer  and  his  fitrep  reflected.  My  second 
WTTP  officer  knocked  it  out  of  the  park” 

“N/A” 

5.  What  skills  should 
an  ASCO  train  on  a 
regular  basis? 

“RIOs,  typing,  GARS/CGRS,  threat  system 
capabilities,  friendly  aircraft  &  ordnance  capabilities” 

“Radio  in  and  out  procedures  to  include  routing, 
safety  of  flight,  and  other  possible  unique  situations 
that  an  officer  might  encounter  at  an  ITX  or  WTI” 
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Table  4  shows  the  SME’s  responses  to  questions  were  they  were  asked  to  rate 
DASC  training  on  a  five  point  Likert  scale,  with  “1”  representing  worst  and  “5” 
representing  best. 


Table  4.  SME  Likert  Responses  Pre-Demonstration 


Question 

SME  1 

SME  2 

2.  How  adequate  is  this  trainina  in  supportin2  skill  acquisition  of 
ASCO? 

3 

1 

h.  How  adequate  is  this  trainin2  in  supportin2  skill  maintenance 
of  ASCO? 

2 

1 

6.  How  would  increasing  the  amount  of  individual  position 
training  improve  the  ability  of  ASCO  to  achieve  and  maintain 
qualification? 

5 

5 

Each  SME  was  given  a  demonstration  of  the  prototype  simulation  produced 
during  this  thesis.  After  the  demonstration  they  were  asked  to  complete  a  second 
questionnaire  regarding  the  demonstration.  Table  5  shows  the  questions  asked  and 
relevant  extracts  from  their  answers.  The  complete  and  answers  questionnaire  can  be 
found  in  the  appendix  B  and  D. 
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Table  5.  SME  Questions  and  Response  Post-Demonstration 


Question 

SME  1 

SME  2 

5.  For  the  purpose  of 
generating  feedback 
reports,  what  metrics 
should  the  system 
capture?  i.e., 
duration  of 
transmission,  failure 
to  brief  safety  of 
flight,  unsafe  routine, 
etc.? 

“-Speech  speed  (words  per  minute) 

-Transmission  duration 

-Response  delay  (how  long  does  it  take  the  for  the 
controller  to  respond) 

-Improper  routing 
-Improper  safety  of  flight 
-Improper  tactics 

-It  should  be  able  to  track  an  individual’s  metrics 
over  time  in  order  to  show  performance  history 
and  improvement/degradation.” 

“Failure  to  brief  safety  of  flight,  co-altitudeing 
aircraft,  routing  around  EN  threats  and/or  gun 
target  lines” 

6.  What  negative 
training  could  a 
training  simulation 
provide? 

“Not  all  aircraft  . . .  should  have  . . .  simulated 
tracks  so  that  that  system  better  replicates  reality. 
Currently,  some  TMS  don’t  have  Link  16  (such 
as  CH-53s,).  There  is  negative  training  value  if  all 
aircraft  have  simulated  tracks  since  they  won’t 
have  tracks  in  reality.  This  could  cause  a  false 
reliance  on  tracks  for  current  position” 

“The  experience  of  controlling  with  live  thinking 
humans.  The  flexibility  of  a  simulator  might 
hinder  a  controller’s  ability  to  reroute  or  pull 
certain  information  from  an  aircraft.” 

7.  Which  attributes 
of  the  prototype 
training  simulation 
were  most  useful? 

“Its  ability  to  recognize/comprehend  speech  and 
take  appropriate  action  based  on  programming” 

“The  most  useful  attributes  were  the  time 
accuracy  of  the  flights  and  missions,  the  ability  to 
hear  different  voices  for  different  pilots,  changes 
of  operational  tempo  if  the  controller  is  doing 
well  or  poor,  and  the  ability  to  integrate  chat 
windows.” 
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Question 

SME  1 

SME  2 

8.  Which  attributes 
of  the  prototype 
training  simulation 
were  least  useful? 

Did  not  provide  an  answer 

“The  least  useful  attributes  included  the  one 
screen  view.  A  controller  would  have  to  click 
through  to  see  the  RIO  board  or  the  chat 
windows.” 

9.  What  additional 
capabilities  should  a 
prototype  training 
system  include  in 
order  to  increase  the 
effectiveness  or 
usability  of  an  ASCO 
IPT? 

“-The  ability  to  split  screen  ATO,  Chat,  and  map. 
-Computer  generated  chat  messages  and 
responses.  This  will  provide  more  realism  and  a 
more  realistic  and  dynamic  training  scenario. 

-It  should  be  able  to  track  an  individual’s  metrics 
over  time  (think  individual  profile)  in  order  to 
show  performance  history  and 
improvement/degradation . 

-It  should  have  a  gui  for  scenario  building  that 
makes  it  easy  and  efficient  to  build  out  a  scenario. 

-It  should  have  the  ability  to  award  points  to  a 
controller  if  there  are  multiple  correct 
statements. .  .think  ‘state  present  position  and 
altitude’  and  ‘go’  at  the  beginning  of  a  check-in” 

“Either  multiple  displays  or  the  ability  to  split 
screen  so  that  a  simulator  felt  like  you  were  in  a 
DASC.  A  controller  learns  a  lot  by  sitting  in  a 
DASC  and  seeing  the  displays  set  up  in  a  realistic 
manner.  The  controller  can  learn  the  system 
scanning  patterns  that  work  best  for  them.” 

10.  What  capabilities 
should  a  prototype 
training  system 
exclude? 

“Currently,  some  TMS  don’t  have  Link  16  (such 
as  CH-53s,).  There  is  negative  training  value  if  all 
aircraft  have  simulated  tracks  since  they  won’t 
have  tracks  in  reality.  This  could  create  a  false 
reliance  on  tracks  for  current  position 
determination  by  controllers.” 

“The  capabilities  in  the  simulator  . . .  [were]  very 
relevant!” 
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Table  4  shows  the  SME’s  responses  to  questions  where  they  were  asked  to  rate 
the  potential  effectiveness  of  a  raining  simulation  for  ASCO  training  on  a  five-point 
Likert  scale,  with  ‘1’  representing  worst  and  ‘5’  representing  best. 


Table  6.  SME  Likert  Responses  Post-Demonstration 


Question 

SME  1 

SME  2 

1.  Would  a  training  simulation  be  effective  at  improving 

ASCO  skills? 

4 

5 

2.  Would  a  training  simulation  be  effective  at  maintaining 
ASCO  skills? 

4 

5 

3.  Could  a  training  simulation  be  used  to  supplement  ASCO 
training? 

4 

5 

4.  Could  a  training  system  be  used  to  replace  a  portion  of 
current  T&R  required  training  events? 

5 

4 

Based  on  this  input,  it  seems  that  there  are  shortfalls  in  ASCO  individual  position 
training.  Further,  the  addition  of  a  training  simulation  could  significantly  improve 
training.  To  improve  the  effectiveness  and  reduce  the  possibility  of  negative  training  the 
SME  suggested  that  the  system  be  able  to  incorporate  multiple  screens  so  the  ASCO 
would  not  have  to  flip  back  and  forth  between  displays  using  the  key  board  and  the 
ASCO  could  practice  using  a  scan  pattern  similar  to  what  would  be  used  in  a  real  DASC. 
Also,  since  the  DASC  is  a  procedural  control  agency,  the  system  should  be  able  to  toggle 
off  the  visual  representation  of  the  aircraft.  Because  flexible  airspace  control  is  an 
important  capability  of  the  DASC,  efforts  should  be  made  to  improve  the  flexibility  of 
control  represented  by  the  system.  For  example,  a  user  should  be  able  to  direct  a  pilot  to 
remain  two  kilometers  north  of  a  given  control  point  or  area. 
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VI.  CONCLUSIONS 


This  work  began  because  we  felt  that  DASC  training  barely  met  the  needs  of  the 
Marine  Corps.  Our  investigations  confirmed  this  impression  and  made  it  clear  that 
current  DASC  training  is  insufficient  to  support  skill  acquisition  and  maintenance.  This 
was  not  due  to  the  problems  with  DASC  personnel  or  leadership,  but  because  the 
personnel  and  time  requirements  to  support  current  training  are  too  high  to  make  regular 
training  feasible  and  airspace  limitations  diminish  the  effectiveness  of  live  exercises.  In 
short,  DASCs  were  too  busy  supporting  other  units’  training  requirements  to  spend 
enough  time  on  their  own. 

We  found  that  one  of  the  major  causes  of  this  was  the  fact  that  there  were  no 
training  aids  available  to  help  DASC  Marines  create,  prepare,  and  execute  their  own 
training  events.  The  amount  of  effort  to  train  two  ASCOs  was  enormous,  generally 
requiring  significant  outlays  of  effort  by  several  Marines  from  the  unit  to  train  those  two. 
Our  research  concluded  that  the  position  of  ASCO  could  benefit  from  a  training 
simulation  which  did  not  currently  exist.  We  found  that  this  limitation  was  primarily  due 
to  the  inherent  complexity  of  DASC  operations  had  precluded  building  a  cheap  and 
readily  available  training  system  previously.  However,  we  felt  that  recent  advances  in 
technology,  primarily  speech  recognition  and  modeling  and  simulation  tools  for 
application  creation,  had  advanced  in  recent  years  to  the  point  where  it  was  possible  to 
create  a  training  device  that  would  allow  ASCOs  to  conduct  worthwhile  training  without 
numbers  of  personnel  spending  substantial  time  and  effort  to  plan  and  execute.  We 
decided  to  attempt  to  build  a  prototypical  simulation  to  provide  training  to  an  ASCO 
without  involvement  of  others. 

In  the  course  of  this  thesis,  we  built  such  a  prototype  for  some  common  events 
that  ASCOs  routinely  perform.  Trainees  use  the  same  voice  commands  to  control 
simulated  aircraft  that  they  would  use  in  the  control  of  actual  aircraft  and  can  operate  the 
system  and  conduct  training  without  the  intervention  of  anyone  else.  As  the  system 
currently  stands,  this  would  provide  no  feedback  to  the  trainee  on  performance,  but  even 
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if  training  required  a  peer  or  senior  Marine  to  observe  the  trainee’s  performance  and 
provide  feedback,  this  is  still  considerably  less  man-hours  than  previously  required. 
Additionally,  the  capability  of  intelligent  tutoring  systems  is  rapidly  expanding  and 
integrating  one  into  the  system  to  allow  training  without  any  other  Marine’s  intervention 
is  definitely  within  the  state  of  the  art. 

To  validate  our  beliefs  in  the  efficacy  of  this  trainer,  we  interviewed  Marines  with 
DASC  experience  and  asked  them  to  comment  on  whether  they  felt  such  a  training 
simulator  would  improve  the  ability  to  train  ASCOs.  They  concurred  that  this  system, 
once  expanded  to  cover  a  larger  percentage  of  DASC  watch  standers’  duties  and 
equipped  with  other  capabilities,  such  as  the  intelligent  tutoring  system,  scenario 
generation  tools,  etc.,  would  greatly  improve  the  ability  to  train  DASC  personnel.  This 
confirmed  our  belief  that  a  useful  training  application  could  be  built  and  distributed 
relatively  cheaply  using  available  technology. 
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VII.  FUTURE  WORK 


Future  work  for  this  project  falls  into  two  main  areas  -  expanding  the  capabilities 
of  the  prototype  and  verifying  its  training  effectiveness. 

A.  EXPANDED  CAPABILITIES 

The  current  prototype  supports  only  a  small  subset  of  the  missions  controlled  by  a 
DASC.  The  system  should  be  expanded  to  handle  all  of  the  different  mission  types,  such 
as: 

•  Airborne  alert  (XCAS) 

•  Forward  air  controller  (airborne)  (FAC  [A]) 

•  Reconnaissance 

•  Refueling 

•  Strike  coordination  and  reconnaissance  (SCAR) 

•  Unmanned  aerial  systems  (UAS) 

•  Downed  aircraft 

•  Casualty  evacuation  (CASEVAC) 

•  Assault  support  coordination  (airborne)  (ASC  [A]) 

•  Very  important  person  (VIP)  transport 

•  Troop  insert 

•  Logistics 

•  Tactical  recovery  of  aircraft  and  personnel  (TRAP) 

•  Pass  and  receive  air  support  request  for  FAV  and  R/W  aircraft 

It  should  also  be  expanded  to  include  models  of  other  DASC  crew  positions  and 
the  external  agencies  with  which  ASCOs  interact  with  regularly.  This  would  provide  the 
ASCO  with  realistic  stimuli  that  could  transfer  to  live  DASC  operations.  These  models 
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should  remain  as  modular  as  possible  to  support  the  eventual  extension  of  the  simulation 
to  include  other  crew  positions. 

Another  important  aspect  that  must  be  added  is  error  tracking  and  feedback.  One 
of  the  shortfalls  of  current  training  identified  by  the  SMEs  was  the  lack  of  metrics  used  to 
track  progress  and  the  qualitative  and  subjective  nature  of  feedback  given  to  trainees. 
Also,  in  order  for  a  training  system  to  be  useful,  there  has  to  be  some  means  to  provide 
corrective  guidance  based  on  their  performance.  Feedback  can  be  provided  in  a  number 
of  different  ways  and  it  may  be  worthwhile  to  incorporated  several  different  methods. 

The  first  method  would  be  to  provide  immediate  feedback.  This  method  would  be 
most  useful  for  new  controllers.  Providing  them  immediate  feedback  will  allow  them  to 
make  immediate  corrections,  prevent  them  from  making  a  habit  of  bad  practices,  and 
allow  them  to  deal  with  a  problem  while  the  experience  is  fresh  in  their  mind. 

The  next  method  would  be  to  give  feedback  at  the  end  of  a  training  session.  This 
would  be  better  for  more  experienced  controllers.  Providing  immediate  feedback  would 
break  their  focus  and  impede  the  learning  process  and  they  are  more  likely  to  be  aware  of 
mistakes  they  are  making  and  remember  specifics  about  the  training  session  to  which  the 
feedback  is  related. 

The  final  method  would  be  to  provide  consequences  to  the  users’  actions.  For 
example,  directing  a  pilot  to  fly  through  a  threat  area  would  result  in  a  downed  aircraft 
situation.  This  method  can  generate  an  emotional  response  which  can  be  particularly 
effective  if  applied  properly.  It  is  important  that  the  user  understand  the  link  between 
their  actions  and  the  consequences.  If  not,  the  lesson  is  infective.  Also,  it  is  important  that 
the  event  not  be  so  traumatic  that  it  damages  the  confidence  of  the  user.  This  could  result 
in  the  user  being  too  timid  to  make  decisions  or  to  causes  to  act  quickly  when  necessary 
and  ultimately  provide  negative  training. 

To  improve  the  usefulness  of  the  performance  tracking  system,  it  may  be 
worthwhile  to  incorporate  a  records  keeping  function.  This  would  allow  a  user  to  track 
their  progression  over  time  and  identify  areas  where  they  could  use  improvement  of 


50 


review  do  to  inactivity.  If  there  was  a  means  to  aggregate  this  information  it  could  be 
used  to  guide  the  development  of  a  squadron  training  plan  and  provide  leadership  with  an 
effective  vision  of  the  command’s  readiness. 

It  may  also  be  useful  to  include  an  instructor’s  interface.  This  would  allow  the 
instructor  to  easily  view  the  progress  of  the  trainees  and  prescribe  training.  Depending  on 
the  robustness  of  this  tool,  it  could  allow  an  instructor  to  review  recorded  training 
sessions  or  even  listen  in  during  training  sessions.  Further,  it  could  provide  the  instructor 
with  the  ability  to  make  on-the-fly  changes  to  an  active  raining  session. 

It  may  be  useful  to  take  this  a  step  further  and  incorporate  an  intelligent  tutoring 
system.  Like  an  instructor,  such  a  system  would  prescribe  training  based  on  past 
performance.  It  may  even  be  able  to  make  dynamic  adjustments  to  in  progress  training 
sessions.  For  example,  if  a  user  was  performing  particularly  well  the  system  would 
automatically  increase  the  difficulty  until  the  user  was  challenged.  On  the  other  hand,  if  a 
user  was  struggling,  the  system  would  automatically  decrease  the  difficulty  to  prevent  the 
user  from  becoming  overwhelmed. 

The  simulation  should  also  be  expanded  to  include  a  scenario  generation  tool. 
Ideally  the  tool  would  operate  in  several  modes.  The  first  mode  would  allow  the  user  to 
input  every  aspect  of  the  simulation.  This  would  be  useful  as  a  mission  rehearsal  tool 
prior  to  the  conduct  of  live  exercises  or  military  operations.  The  ASCO  would  be  able  to 
practice  controlling  in  the  airspace  in  its  actual  configuration  and  become  familiar  with 
the  participating  units  and  their  call  signs. 

The  second  mode  would  allow  the  user  to  input  a  portion  of  the  scenario  and 
randomly  generate  the  next.  This  would  be  useful  if  a  user  needed  to  practice  certain  aspects 
of  controlling,  such  as  downed  aircraft  procedures,  but  other  specific  information,  such  as 
radio  frequency  code  words,  was  unimportant.  This  would  allow  for  focused  training  without 
the  overhead  of  building  an  entire  scenario.  The  third  mode  would  create  a  completely 
random  scenario  and  require  no  user  input.  His  mode  would  be  useful  to  support  day  to  day 
training  and  would  prevent  ASCO  from  becoming  too  familiar  with  a  given  scenario.  The 

scenario  generation  tool  should  have  a  user  interface  to  facilitate  ease  of  use. 
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B.  TRAINING  EFFECTIVENESS  EVALUATION 


The  system  should  be  tested  on  several  levels.  A  usability  study  should  be 
conducted  to  see  if  users  find  interacting  with  the  system  cumbersome.  If  the  system  is 
too  difficult  to  use,  is  too  error  prone,  or  does  not  meet  the  user’s  expectations  in  some 
other  way,  it  will  not  be  used. 

A  transfer  of  training  study  should  be  conducted.  It  is  important  that  the  use  of  the 
system  actually  result  in  improved  performance.  If  not,  then  it  is  just  a  waste  of  resources 
and  another  administrative  burden  for  the  unit.  Transfer  of  training  studies  should  be 
conducted  at  several  different  levels,  examining  whether  they  system  provides  benefits  to 
users  across  the  experience  spectrum.  Novices  need  to  perform  different  tasks  to  become 
advanced  beginners  than  proficient  operators  need  to  perform  to  become  experts. 
Understanding  how  the  system  should  be  used  to  maximize  improvements  for  learners  at 
different  levels  could  provide  valuable  information  on  how  to  structure  both  a  training 
and  acquisition  program  for  such  a  system. 

At  a  higher  level,  analysis  should  examine  how  the  addition  of  such  a  system 
effects  on  ASCO  training  progression  and  how  it  affects  the  readiness  of  a  squadron.  If 
the  addition  of  a  training  simulation  improves  skill  acquisition  and  maintenance,  the  time 
required  for  an  ASCO  to  achieve  qualifications  could  be  reduced  resulting  in  more 
qualified  ASCOs  and  improving  the  readiness  of  the  squadrons.  Likewise,  if  the  time  and 
effort  to  produce  training  events  is  reduced,  it  may  be  possible  to  require  operators  to 
perform  more  of  them,  resulting  in  a  better  trained  group. 

Finally,  it  is  important  to  make  sure  that  we  are  investing  the  right  system.  The 
prototype  training  simulation  produce  during  this  thesis  represent  only  a  single 
configuration.  One  of  the  next  steps  would  be  to  conduct  an  analysis  of  alternatives  at  the 
system  level.  It  may  be  that  a  system  that  does  not  use  voice  recognition,  but  allows  a 
facilitator  to  manually  update  routes  and  provides  the  facilitator  with  responses  to  read  to 
the  user,  is  a  more  suitable  option. 
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APPENDIX  A.  SME  1  PRE -DEMONSTRATION  QUESTIONNAIRE 


Questions  for  SME  on  ASCO  IPT 

The  scope  of  the  questions  below  is  restricted  to  the  individual  position  training  of  air  support  control  officers.  It  is 
not  focused  on  other  DASC  positions  individual  position  training  except  as  possible  extensions  for  future  work.  It 
does  not  focus  on  crew  training  except  in  respect  to  the  improvements  that  may  accrue  as  a  result  of  improved 
individual  performance. 

As  a  DASC  SME.  please  respond  to  the  following  questions  in  context  of  DASC  operations  and  training: 

1.  Dose  a  gap  exists  for  air  support  control  officer's  individual  position  training?  If  so  describe  the  gap. 

Yes,  we  do  not  have  a  procedural  control  publication  in  the  Marine  Corps.  Instead  we  have  an  individual 
position  training  handbook  maintained  by  the  Air  Support  Training  Section  at  Marine  Corps  Communications 
Electronic  School.  While  the  IPT  handbook  does  support  general  procedural  control  techniques,  it  is  not 
authoritative,  since  it  is  not  a  MCRP.  Additionally,  because  it  is  locally  maintained,  it  is  at  risk  of  being  changed  to 
align  to  the  personal  experiences  of  whomever  maintains  it. 

Once  out  of  MOS  schools,  "Individual"  position  training  does  not  truly  exist.  A  significant  number  2000 
level  T&R  events  (individual  training  events),  actually  require  a  crew  and  a  white  cell  to  execute.  Because  the 
many  of  the  2000  level  training  events  require  a  crew  and  a  white  cell  to  execute,  training  opportunities  are  arise 
when  DASC  Drills  are  scheduled  not  based  on  the  individual's  training  requirement  for  progression.  This  limits  the 
number  of  reps  that  an  individual  can  get  in  a  given  year,  limiting  their  proficiency  as  a  result. 

Additionally,  there  is  not  standardization  among  the  MASS  units  for  how  each  of  these  events  is  designed. 
In  my  mind,  ever  officer  sitting  a  Ctrl  2000  event  should  be  able  to  run  through  the  same  scenario  or  set  of 
scenarios  for  a  given  T&R  event.  Instead,  for  each  DASC  drill  mentioned  above,  a  half  bit  scenario  is  developed  and 
executed,  usually  without  regard  to  what  the  individual  is  required  to  do  in  order  to  accomplish  the  T&R  event. 

2.  What,  if  any,  individual  position  training  is  conducted  prior  to  a  DASC  drills?  There  are  very  few  T&R  events 
that  are  conducted  prior  to  DASC  Drills.  This  is  partly  due  to  the  factors  mentioned  above,  but  also  do  to  the  high 
op  tempo  of  the  Squadrons.  DASC  Drills  are  typically  crammed  in  between  exercises  that  squadron  is  supporting, 
and  because  most  2000  and  3000  T&R  events  inherently  require  a  crew,  a  whitecell,  and  a  qualified  evaluator  it 
rarely  occurs. 

3.  What,  if  any,  individual  position  training  is  conducted  prior  to  live  exercises  or  operations?  Prior  to  combat 
operations,  the  squadrons  do  a  pretty  good  job  of  running  DASC  drills  using  a  scenario  that  mimics  the  theater  of 
operations.  But  this  is  rarely  "individual"  for  the  reasons  mentioned  above. 

4.  What,  if  any,  individual  position  training  is  conducted  between  these  periods  for  the  purpose  of  skill 
maintenance?  For  each  type,  if  any,  please  Identify  the  following: 

a.  How  often  (In  months)  does  this  training  occur?  We  tried  to  do  a  DASC  drill  once  a  quarter  but  this 
was  driven  by  the  squadron's  op  tempo. 

b.  How  is  this  training  evaluated  (sat/unsat,  grades,  etc.)?  Performance  evaluation  is  captured  on  a 
MACCS  training  form,  0-4  scale  with  zero  being  unsat  and  4  representing  mastery.  A  student  must  score  at  least  a 
3  (proficient)  in  order  to  receive  credit  for  the  T&R  event.  However,  this  is  purely  a  qualitative  evaluation  by 
evaluator.  Rarely  are  an  actually  metrics  included  in  the  performance  evaluation. 

c.  List  the  types  feedback  which  is  provided  to  the  trainee  (real-time,  after  action  review,  none,  etc.)? 

After  every  crew  executes  the  drill,  there  is  a  quick  hot  wash  by  the  evaluators.  This  is  purely  qualitative. 
No  actual  metrics  are  collected  and  used  to  measure  performance.  Additionally  each  evaluator  is 
supposed  to  debrief  the  individual  they  were  evaluating  on  their  performance  in  greater  detail  providing 
areas  for  improvement. 
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Questions  for  SME  on  ASCO  IPT 


d.  Are  records  of  this  training  maintained?  Yes  on  the  M ACCS  Training  Forms 

e.  How  are  these  records  used  to  inform  future  training  of  individual  ASCO?  WTTP  is  supposed  to  look  at 
what  training  events  an  individual  has  completed  and  what  training  events  they  need  to  complete  in  order  to  get 
qualified,  and  then  build  drills  to  meet  the  requirements.  As  I  mentioned  previously,  very  few  WTTP  officers  have 
the  mental  agility  to  make  this  happen.  Typically  a  scenario  is  built,  individuals  sit  position  on  a  crew,  and 
evaluators  must  create  the  training  in  real  time  in  order  for  the  individual  to  get  the  X. 

f.  How  are  these  records  used  to  inform  the  Squadron  training  plan?  I  was  fortunate  that  my  last  WTTP 
officer  who  understood  my  vision.  It's  quite  simple.  We  must  have  a  certain  amount  of  individuals  qualified  in 
certain  positions  per  the  T&R  manual  for  DRRS  reporting.  The  WTTP  officer  was  tasked  with  identifying  who 
needed  to  get  which  qualification  in  order  to  make  all  of  our  crews  qualified.  Once  that  was  accomplished,  he  had 
to  identify  what  training  requirements  each  individual  needed  in  order  to  receive  position  qualification.  Based  on 
the  requirements  of  the  individuals,  WTTP  was  tasked  with  coming  up  with  individualized  training  plans  and  a 
squadron  training  plan  that  supported  satisfaction  of  each  individual's  requirement.  This  was  too  much  work  for 
my  first  WTTP  officer  and  his  fitrep  reflected.  My  second  WTTP  officer  knocked  it  out  of  the  park. 


g.  How  adequate  is  this  training  in  supporting  skill  acquisition  of  ASCO? 


Not  at  all  adequate 

Adequate  for 
acquiring  some 
skills 

Complete  skill 
acquisition 

1 

2 

3 

4 

5 

H.  How  adequate  is  this  training  in  supporting  skill  maintenance  of  ASCO? 


Not  at  all  adequate 

Adequate  for 
maintaining  some 
skills 

Complete  skill 
maintenance 

1 

2 

3 

4 

5 

5.  What  skills  should  an  ASCO  train  on  a  regular  basis?  RIOs,  typing,  GARS/CGRS,  threat  system  capabilities, 
friendly  aircraft  &  ordnance  capabilities 


6.  How  would  increasing  the  amount  of  individual  position  training  improve  the  ability  of  ASCO  to  achieve  and 
maintain  qualification? 


No  Improvement 

Neither  improve 
nor  detract 

Complete 

Improvement 

1 

2 

3 

4 

1 

***Reps  without  feedback  are  just  reps.  You  can  get  really  good  at  doing  the  wrong  thing  this  way.  Reps  need 
to  have  feedback  based  on  pre-determined  qualitative  and  quantitative  standards  for  performance.  *** 
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APPENDIX  B.  SME  1  POST-DEMONSTRATION  QUESTIONNAIRE 


Questions  for  SME  on  ASCO  IPT 


The  scope  of  the  questions  below  is  restricted  to  the  individual  position  training  of  air  support  control 
officers.  It  is  not  focused  on  other  DASC  positions  individual  position  training  except  as  possible 
extensions  for  future  work.  It  does  not  focus  on  crew  training  except  in  respect  to  the  improvements 
that  may  accrue  as  a  result  of  improved  individual  performance. 

Considering  the  DASC  simulation  demonstration  that  you  just  received,  as  a  DASC  SME,  answer  the 
following  questions  in  relation  to  DASC  training: 


1.  Would  a  training  simulation  be  effective  at  improving  ASCO  skills? 


No  Improvement 

Neither  improve 
nor  detract 

Completely 

Sufficient 

i 

2 

3 

4 

5 

2.  Would  a  training  simulation  be  effective  at  maintaining  ASCO  skills? 


Not  effective 

Neither  improve 
nor  detract 

Extremely 

Effective 

i 

2 

3 

4 

5 

3.  Could  a  training  simulation  be  used  to  supplement  ASCO  training? 


Not  possible 

Likely 

Definitely 

i 

2 

3 

4 

5 

4.  Could  a  training  system  be  used  to  replace  a  portion  of  current  T&R  required  training  events? 


Not  possible 

Likely 

Definitely 

i 

2 

3 

4 

5 

5.  For  the  purpose  of  generating  feedback  reports  what  metrics  should  the  system  capture?  i.e. 
duration  of  transmission,  failure  to  brief  safety  of  flight,  unsafe  routine,  etc. 

-Speech  speed  (words  per  minute) 

-Transmission  duration 

-Response  delay  (how  long  does  it  take  the  for  the  controller  to  respond) 

-Improper  routing 
-Improper  safety  of  flight 
-Improper  tactics 

-It  should  be  able  to  track  an  individual's  metrics  over  time  in  order  to  show  performance 
history  and  improvement/degradation. 
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Questions  for  SME  on  ASCO  IPT 


6.  What  negative  training  could  a  training  simulation  provide?  Not  all  aircraft  for  should  have  a 
simulated  tracks  so  that  that  system  better  replicates  reality.  Currently,  some  TMS  don't  have  Link  16 
(such  as  CH-53s,).  There  is  negative  training  value  if  all  aircraft  have  simulated  tracks  since  they  won't 
have  tracks  in  reality.  This  could  cause  a  false  reliance  on  tracks  for  current  position. 

7.  Which  attributes  of  the  prototype  training  simulation  were  most  useful? 

Its  ability  to  recognize/comprehend  speech  and  take  appropriate  action  based  on  programming. 

8.  Which  attributes  of  the  prototype  training  simulation  were  least  useful? 

9.  What  additional  capabilities  would  should  a  prototype  training  system  include  in  order  to  increase 
the  effectiveness  or  usability  of  an  ASCO  IPT? 

-The  ability  to  split  screen  ATO,  Chat,  and  map. 

-Computer  generated  chat  messages  and  responses.  This  will  provide  more  realism  and  a  more  realistic 
and  dynamic  training  scenario. 

-It  should  be  able  to  track  an  individual's  metrics  over  time  (think  individual  profile)  in  order  to  show 
performance  history  and  improvement/degradation. 

-It  should  have  a  gui  for  scenario  building  that  makes  it  easy  and  efficient  to  build  out  a  scenario. 

-It  should  have  the  ability  to  award  points  to  a  controller  if  there  are  multiple  correct  statements-.think 
"state  present  position  and  altitude"  and  "go"  at  the  beginning  of  a  check-in. 

10.  What  capabilities  should  a  prototype  training  system  exclude?  Not  all  aircraft  for  should  have  a 
simulated  tracks  so  that  that  system  better  replicates  reality.  Currently,  some  TMS  don’t  have  Link  16 
(such  as  CH-53s,).  There  is  negative  training  value  if  all  aircraft  have  simulated  tracks  since  they  won't 
have  tracks  in  reality.  This  could  create  a  false  reliance  on  tracks  for  current  position  determination  by 
controllers. 
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APPENDIX  C.  SME  2  PRE-DEMONSTRATION  QUESTIONNAIRE 


Questions  for  SME  on  ASCO  IPT 


The  scope  of  the  questions  below  is  restricted  to  the  individual  position  training  of  air  support  control 
officers.  It  is  not  focused  on  other  OASC  positions  individual  position  training  except  as  possible 
extensions  for  future  work.  It  does  not  focus  on  crew  training  except  in  respect  to  the  improvements 
that  may  accrue  as  a  result  of  improved  individual  performance. 

As  a  DASC  SME,  please  respond  to  the  following  questions  in  context  of  DASC  operations  and  training: 

1.  Dose  a  gap  exists  for  air  support  control  officer's  individual  position  training?  If  so  describe  the  gap. 

Yes,  right  now  the  only  individual  training  occurs  at  MOS  school.  There  is  very  minimal  training  that 
focuses  on  an  individual  controller.  The  most  individual  training  a  controller  gets  is  practicing  Radio  in 
and  out  (RIO)  with  other  officers. 

2.  What,  if  any,  individual  position  training  is  conducted  prior  to  a  DASC  drills? 

Like  I  stated  above,  the  only  individual  position  training  is  practicing  RIOs  with  another  officer  (most  of 
the  time  this  is  limited). 

3.  What,  if  any,  individual  position  training  is  conducted  prior  to  live  exercises  or  operations? 

Very  minimal,  if  there  is  time  a  more  senior  officer  might  practice  and  mentor  a  junior  officer.  Most  of 
the  time  more  senior  officers  are  busy  planning  and  prepping  for  exercises  or  operations,  so  very  little 
time  if  any  is  spent  on  individual  training.  Most  of  the  training  is  conducted  at  the  crew  level,  making 
sure  the  crew  is  ready  to  operate  as  a  whole. 

4.  What,  if  any,  individual  position  training  is  conducted  between  these  periods  for  the  purpose  of  skill 
maintenance?  For  each  type,  if  any,  please  identify  the  following: 

a.  How  often  (in  months)  does  this  training  occur? 

Every  4  months...  at  bestl 

b.  How  is  this  training  evaluated  (sat/unsat,  grades,  etc.)? 

None 

c.  List  the  types  feedback  which  is  provided  to  the  trainee  (real-time,  after  action  review, 
none,  etc.)? 

None 

d.  Are  records  of  this  training  maintained? 

No 

e.  How  are  these  records  used  to  inform  future  training  of  individual  ASCO? 

N/A 

f.  How  are  these  records  used  to  inform  the  Squadron  training  plan? 

N/A 
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Questions  for  SME  on  ASCO  IPT 


g.  How  adequate  is  this  training  in  supporting  skill  acquisition  of  ASCO? 


Not  at  all 
adequate 

Adequate  for 
acquiring  some 
skills 

Complete  skill 
acquisition 

I 

2 

3 

4 

5 

H.  How  adequate  is  this  training  in  supporting  skill  maintenance  of  ASCO? 


Not  at  all 
adequate 

Adequate  for 
maintaining  some 
skills 

Complete  skill 
maintenance 

1 

2 

3 

4 

5 

5.  What  skills  should  an  ASCO  train  on  a  regular  basis? 

Radio  in  and  out  procedures  to  include  routing,  safety  of  flight,  and  other  possible  unique  situations  that 
an  officer  might  encounter  at  an  ITX  or  WTI. 


6.  How  would  increasing  the  amount  of  individual  position  training  improve  the  ability  of  ASCO  to 
achieve  and  maintain  qualification? 


No  Improvement 

Neither  improve 
nor  detract 

Complete 

Improvement 

1 

2 

3 

4 

1 
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APPENDIX  D.  SME  2  POST-DEMONSTRATION  QUESTIONNAIRE 


Questions  for  SME  on  ASCO  IPT 


The  scope  of  the  questions  below  is  restricted  to  the  individual  position  training  of  air  support  control 
officers.  It  is  not  focused  on  other  DASC  positions  individual  position  training  except  as  possible 
extensions  for  future  work.  It  does  not  focus  on  crew  training  except  in  respect  to  the  improvements 
that  may  accrue  as  a  result  of  improved  individual  performance. 


Considering  the  DASC  simulation  demonstration  that  you  just  received,  as  a  DASC  SME,  answer  the 
following  questions  in  relation  to  DASC  training: 

1.  Would  a  training  simulation  be  effective  at  improving  ASCO  skills? 


No  Improvement 

Neither  improve 
nor  detract 

Completely 

Sufficient 

1 

2 

3 

4 

1 

2.  Would  a  training  simulation  be  effective  at  maintaining  ASCO  skills? 


Not  effective 

Neither  improve 
nor  detract 

Extremely 

Effective 

1 

2 

3 

4 

1 

3.  Could  a  training  simulation  be  used  to  supplement  ASCO  training? 


Not  possible 

Likely 

Definitely 

1 

2 

3 

4 

1 

4.  Could  a  training  system  be  used  to  replace  a  portion  of  current  T&R  required  training  events? 


Not  possible 

Likely 

Definitely 

1 

2 

3 

T 

5 

5.  For  the  purpose  of  generating  feedback  reports  what  metrics  should  the  system  capture?  i.e. 
duration  of  transmission,  failure  to  brief  safety  of  flight,  unsafe  routine,  etc. 

Failure  to  brief  safety  of  flight,  co-altitudeing  aircraft,  routing  around  EN  threats  and/or  gun  target  lines 

6.  What  negative  training  could  a  training  simulation  provide? 

The  experience  of  controlling  with  live  thinking  humans.  The  flexibility  of  a  simulator  might  hinder  a 
controllers  ability  to  reroute  or  pull  certain  information  from  an  aircraft. 

7.  Which  attributes  of  the  prototype  training  simulation  were  most  useful? 

The  most  useful  attributes  were  the  time  accuracy  of  the  flights  and  missions,  the  ability  to  hear 
different  voices  for  different  pilots,  changes  of  operational  tempo  if  the  controller  is  doing  well  or  poor, 
and  the  ability  to  integrate  chat  windows. 
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Questions  for  SME  on  ASCO  IPT 


8.  Which  attributes  of  the  prototype  training  simulation  were  least  useful? 

The  least  useful  attributes  included  the  one  screen  view.  A  controller  would  have  to  click  through  to  see 
the  RIO  board  or  the  chat  windows. 

9.  What  additional  capabilities  would  should  a  prototype  training  system  include  in  order  to  increase 
the  effectiveness  or  usability  of  an  ASCO  IPT? 

Either  multiple  displays  or  the  ability  to  split  screen  so  that  a  simulator  felt  like  you  were  in  a  DASC.  A 
controller  learns  a  lot  by  sitting  in  a  DASC  and  seeing  the  displays  set  up  in  a  realistic  manner.  The 
controller  can  learn  the  system  scanning  patterns  that  work  best  for  them. 

10.  What  capabilities  should  a  prototype  training  system  exclude? 

The  capabilities  in  the  simulator  was  very  relevant! 
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APPENDIX  E.  IRB  DETERMINATION 


a 19s.  7on.uo~n~  do -k/ 

NPS  IRB 

HSR  Determination  Checklist 
last  updated:  2-28-1 7 

Instructions:  This  form  is  to  be  completed  by  the  IRB  Chair  or  Vice-Chair  when  providing  an  official  IRB  determination  on 
whether  a  proposed  activity  meets  the  federal  definition  of  research  with  human  subjects  according  to  32  CFR  219.  After 
completing  the  form  provide  to  the  IRB  Administrator.  The  IRB  Administrator  will  notify  the  investigators  and  file  electronically. 

Title  of  the  Activity:  SME  Evaluation  of  DASC  Training  Simulation  Prototype 


Department:  OR/MOVES 


Principal  Investigator:  Lee  Sciarini 


Co-Investigators: 


Perry  McDowell 


Kathleen  Haggard 

Student  Researchers: 

Determination  Criteria 

1 .  Is  the  activity  research?  For  the  activity  to  be  research  both  (a)  and  (b)  must  be  answered  in  the  affirmative. 
(a)  Is  the  activity  a  systematic  investigation? 


Yes  No 

B  □ 


(b)  Is  the  activity  designed  to  develop  or  contribute  to  generalizable  knowledge? 


□ 


2. 


Does  the  activity  involve  the  use  of  human  subjects?  For  the  activity  to  involve  human  subjects  (a)  and  (b)  or  (c)  must  be 


answered  in  the  affirmative. 

(a)  Is  the  activity  designed  to  collect  information  about  a  living  individual? 


Yes  No 

□  □ 


(b)  Does  the  activity  involve  interaction  with  a  person  or  persons? 


□  □ 


(c)  Does  the  activity  involve  the  use  of  pre-collected  information  that  is  both  private  and  personally  identifiable?  □  □ 


IRB  Determination 


The  attached  activity  involves  human  subjects  and  requires  IRB  review  and  NPS  President  approval. 


Yes  No 

□  El 


IRB  Chair/Vice  Chair: 


K.  J.  Euske 


Digitally  signed  by  K.  J.  Euske 
ON:  cn=K.  J.  Euske,  o=NPS,  ou=GSBPP, 
email=kjeuske@nps.edu,  c=US 
Date:  2017.04.25  09:43:17  -07'00‘ 


Date: 
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